OER (Optimize Edge Routing) Phases

If you are new to OER I suggest to stop now and read my Introduction to OER first. This will give you an idea what OER is and why you might want to use it. Reading about the different OER phases will be very confusing if you don’t know the basics. After reading the introduction it’s best to start with a basic configuration first so you’ll see how it works. Having said that, let me show you the different phases:

• OER Profile Phase
• OER Measure Phase
• OER Apply Policy Phase
• OER Control Phase
• OER Verify Phase

These 5 phases always loop around. OER will start with the profile phase and then moves on to the measure, apply policy, control and verify phase. After the verify phase it will go back to the profile phase and this cycle will keep on going. Now let’s take a closer look at the different phases:

OER Profile Phase

Depending on the size of your network you might have hundreds or thousands of routes in the RIB (Routing Information Base). Optimize edge routing means that we will prefer some traffic over other traffic so we have to select a number of routes from the RIB that we want to optimize.  There are a couple of ways how we can do this:

• We can automatically learn traffic flows that experience performance issues.
• It’s also possible to manually configure different traffic classes.

So what is a “traffic class” exactly? It can be something simple like a prefix but it’s also possible to use a prefix in combination with a port number. Border routers will learn about traffic classes and report this to the master controller. These traffic classes are then stored in a special table called the Monitored Traffic Classes (MTC) table. This MTC has a limited capacity so by default OER will perform prefix aggregation. It will aggregate (summarize) all prefixes to a /24 by default. The MTC will store 100 prefixes but this is something we can change if we want to.

Automatically learning of traffic classes is done by using the top talkers feature of NetFlow. You don’t have to configure NetFlow yourself, it is done for you automatically when you enable OER.

OER doesn’t check for traffic classes non-stop but it works with a schedule:

OER will learn in the monitor period which is 5 minutes by default. Once this period is over it will go into “sleep mode” (120 minutes) and it will save the information on the prefixes that it has learned. These timers might be ok for production networks but you might want to speed it up for your labs.

OER Measure Phase

Once we know what traffic classes we want to optimize (either by learning or configuring them ourselves) OER will measure the performance of each traffic class. There are two methods how we can do this:

• Passive monitoring
• Active monitoring

Passive monitoring uses Netflow and interface counters on the border routers. OER will measure the following performance metrics using NetFlow:

• Delay: The average delay of TCP flows for a prefix. It will keep track of TCP segments and the returning TCP ACK to calculate round-trip response time (RTT).
• Packet Loss: By keeping track of TCP sequence numbers OER can measure packet loss.
• Reachability: OER will keep track of TCP SYNs that have been sent without receiving a TCP ACK response.
• Throughput: The total number of bytes and packets for each traffic class in a certain amount of time. This is one of the metrics that can be used for non-TCP traffic.

Active monitoring uses IP SLA to emulate the traffic class and discover performance metrics. Our border routers will report their information to the master controller, the master controller will store the performance metrics together with the traffic classes in the MTC. OER can collect the following performance metrics thanks to IP SLA:

• Delay (same as above).
• Reachability (same as above).
• Jitter: A variation in delay causes jitter. OER will send multiple packets to the destination and measure the delay between them.
• MOS: The Mean Opinion Score is used to represent voice quality on a scale from 1 to 5. 1 = terrible voice quality, 5 = best voice quality

For each traffic class or interface that OER monitors it will check the current state and it does this by using different states:

• Default: traffic classes in the default state are not controller by OER. When a traffic class is added to the MTC it will be in the default state. You will see traffic classes go into and out the default state depending on measurement results and the policy that you have configured.
• Choose Exit: This is where OER compares performance metrics against the configured policy for the traffic class. OER will prefer to keep the current “exit path” for a certain traffic class but when it exceeds its current policy the master controller will start looking for another exit path.
• Holddown: A traffic class will be in the holddown state when the master controller tells the border router  to use active probes (IP SLA) to monitor the traffic class. Performance metrics are collected until the holddown timer expires.
• In-Policy: After comparing the performance metrics against the policy and an exit path has been selected, the traffic class will be in-policy. This means the traffic class doesn’t exceed our policy…the master controller will keep monitoring the traffic class but no action is taken unless the periodic timer expires or when the traffic class performance metrics exceeds the policy.
• Out-of-Policy (OOP): When there are no exit paths that conform to the policy, the traffic class will go out-of-policy. The backoff timer controls whether a traffic class can leave this state or not and every time a traffic class goes to the out-of-policy state this timer will increase. The backoff timer will reset when the traffic class goes to the in-policy state. When all exit paths are out-of-policy the master controller can select the best exit path available.

OER Apply Policy Phase

On the master controller we will configure a policy with certain thresholds for our traffic classes to define “acceptable performance”. OER will compare the performance metrics with the policy that we created. When the performance metrics exceed the threshold in our policy the traffic class will go OOP (Out of Policy). OER will keep comparing these results to see if there are changes in traffic flows. There are two types of policies for OER:

• Traffic class policies
• Link policies

Traffic class policies are configured for prefixes or applications. Link policies are used for exit or entrance links at the edge of our network. For example we can configure OER that traffic towards destination 4.4.4.0/24 should never have a higher delay than 150 ms. As soon as the delay is higher than 150 ms, OER will look for a better exit path so that we conform to the policy.

OER Control Phase

The control phase is where the action happens. When OER has decided that some traffic classes do not confirm to the policy it will change routing on the border routers. It can do this by injecting static routes, BGP routes, changing the BGP local preference, changing route metrics and/or using policy based routing. As a result some traffic classes will have a different exit path. OER will initiate a route change when any of the following occurs:

• Traffic out-of-policy: when the performance metrics exceed the configured policy.
• Exit link out of policy: when the link (interface) exceeds link utilization or loses connectivity.
• Periodic timer expires: when OER is configured for “best mode” the master controller will start looking for the best exit path for the traffic class.

When a traffic class is defined only by a prefix then OER can use static route or BGP route injection. These changes are network-wide since it will affect all routers in the network. Before injecting routes, OER will verify if you already have this route in your BGP table or configured as a static route…if not, it is impossible to inject the route. OER requires a parent route in order to inject routing information. This makes sense because OER is not a routing protocol…if it would just send traffic in a certain direction you might end up blackholing traffic or creating routing loops. Make sure you have configured a floating static route or have a valid entry in the BGP table.

OER is not a routing protocol…make sure you have a valid parent route if you want to send traffic another way.

Traffic classes that have a prefix AND a port number can not be influenced by using route injection When this occurs the changes are not network-wide but device-specific. In this case OER will use policy based routing to influence routing.

When you use PBR to influence routing, the border routes have to be one hop away…physically or otherwise by using a GRE tunnel.

OER Verify Phase

After making changes and sending traffic in different directions, OER will verify that the traffic is optimized and using a different exit interface. OER will collect the resulting statistics to verify that the changes bring the traffic classes “in policy”. The master controller will verify this by watching netflow information from the interface of the new exit path and ignoring netflow information from the old interface.

That’s all I have for now. I hope this helps you to understand OER better. If you have any questions just leave a comment!

OER Variance to solve Policy Conflicts

When you configure a policy for OER you can choose what metrics are most important to you (packet loss, delay, jitter, etc.). For example you can configure a policy that will choose an exit path based on the following criteria:

1. Lowest Delay
2. Lowest Jitter
3. Lowest Packet Loss

Now imagine we have two exit paths with the following metrics:

1. Exit A: Delay is 80ms, jitter is 5ms and packet loss is 2000ppm.
2. Exit B: Delay is 90ms, jitter is 3ms and packet loss is 1500ppm.

If we select exit A based on the lowest delay only then we don’t make the best choice since the jitter and packet loss of exit A are worse than exit B. By using the variance we can ensure that other exit paths fall within the “acceptable” criteria for a certain metric. This allows us to choose an exit path based on more than one metric.

Now let’s say I configure a variance of 15% for delay. Exit path A has a delay of 80 ms. All exit paths that fall within 80 ms + 15% will be considered equal when it comes to delay. 80ms + 15% = 92 ms. The delay for exit B (90 ms) is lower than 92 ms so exit A and B are seen as “equal” when it comes to delay. This means we will have to look at the second metric (jitter).

Does this make sense? Let’s take a look at an actual policy with 3 exit paths:

• Exit A: Delay 100 ms, jitter 5ms, packet loss 2000ppm.
• Exit B: Delay 150 ms, jitter 6ms, packet loss 1900ppm.
• Exit C: Delay 110 ms, jitter 3ms, packet loss 1900ppm.

MC#show run | sec oer-map
oer-map VARIANCE 10
 match traffic-class prefix-list MYPREFIXES
 set resolve delay priority 1 variance 20
 set resolve loss priority 2 variance 10
 set resolve jitter priority 3 variance 15

Let me describe the policy above:

• Priority 1 is delay but we have a variance of 20%.
• Priority 2 is loss with a variance of 10%.
• Priority 3 is jitter with a variance of 15%.

Now which exit path are we going to use? Let me walk you through the process:

1. First we take a look at the lowest delay, that is exit A. The variance however is 20%. 100 ms + 20% = 120 ms.
2. Exit C is still in the race because 110 ms is lower than 120 ms. Exit B has a too high delay so it will not be used.
3. The second priority is packet loss so we’ll compare exit A and B. Exit C has a packet loss of 1900 ppm (packet per million). There is a variance of 10% however. 1900 ppm + 10% = 2090 ppm. Exit A has a packet loss of 2000 which is lower than 2090 ppm so exit A is still in the race…
4. We’ll have to compare the last metric which is jitter. Exit C has 3 ms jitter and exit A has 5 ms jitter. As a result we will select exit C as the most optimal exit path.

That’s how OER will select the most optimal path when using variance. I hope this has been helpful to you! If you have any questions just leave a comment.

OER (Optimized Edge Routing) Timers for Labs

One of the annoyances of working with OER (Optimized Edge Routing) in labs is that it is based on timers for different actions. This can be very annoying when you are doing labs since it means you will be staring for minutes (or hours) at your console screen waiting for OER to take any action.

We can however optimize the OER timers to make it learn and react faster, in this lesson I want to give you an overview of the timers we can change.

First of all you might want to consider changing the load interval of your external interfaces on the border routers. By default the load interval is calculated each 5 minutes which is an eternity when you are doing labs. We can change this to a minimum of 30 seconds like this:

BR(config)#interface serial 0/0
BR(config-if)#load-interval 30

30 seconds is the lowest value we can use, configure it using the load-interval command.

The next step is to take a look at the actual OER timers. For the learning process there are two important timers:

• periodic-interval
• monitor-period

The learning process for OER has a periodic-interval and a monitor-period. The periodic-interval is a sleep timer that tells OER how often to start the learning process and how to long to sleep and retain the information that it has learned. By default this is 2 hours which is way too long for a lab. If you keep it at the default you might find yourself restarting the OER master cotroller more often than you wish for.

The monitor-period defines how long we will learn the different traffic flows. By default this is 5 minutes but we can reduce it. Simply said…OER will learn for a period and then go back to sleep, wake up after a while to learn again and so on. This is how we can change these timers:

MC(config)#oer master
MC(config-oer-mc)#learn
MC(config-oer-mc-learn)#periodic-interval 1
MC(config-oer-mc-learn)#monitor-period 1

I will set both timers to 1 minute. This makes it suitable for a lab.

There are two more OER timers that you should adjust:

• backoff
• holddown

The backoff timer tells OER that once a traffic flow is out of policy, the master controller will wait for a certain amount of time before it tries to find another exit path. This backoff timer tells the master conroller how long to wait and it increases each time the master controller is unable to find an optimal exit path. We use this backoff timer so the master controller gets more time to find an exit path after each time it is unsuccessful.

We have to configure a minimum and maximum time value and also a step value. By default the minimum timer is 300 seconds, maximum is 3000 seconds and the step value is 300 seconds. This means that the first time the master controller will have 300 seconds to find an exit path, if it is unsuccessful, the next time it will add the step time (300 seconds) so it will have 600 seconds. If it’s still unsuccessful the next time it will take 900 seconds, 1200, 1500 and so on….up to a maximum of 3000 seconds.

The holddown timer tells OER how long a new exit path must be used after it went out of policy before we can look for a new best exit path. We do this to prevent flapping between different exit paths. By default this is 300 seconds and on some routers you can reduce it to 90 seconds.

Here is how to change the backoff and holddown timer:

MC(config)#oer master
MC(config-oer-mc)#holddown 90
MC(config-oer-mc)#backoff 90 180 90

This will change the holddown timer to 90 seconds and the min and max backoff timer to 90 seconds.

That’s all I wanted to share with you for now, I hope these changes will help speed up your OER labs. If you have any questions feel free to ask!

OER (Optimized Edge Routing) Basic Configuration

In this lesson I want to show you a basic configuration of OER (Optimized Edge Routing). If you have no idea what OER is or why you want to use this, take a look at my introduction to OER first.

OER is not a simple topic and the configuration can become quite complex because of all the policies. In this example i’m going to walk you through a simple scenario where we configure an MC (Master Controller) and BR (Border Router). Let me show you the topology:

There are quite some routers so let me explain this topology to you:

• R1,R2 and R3 belong together, the serial links on R3 are the “edge” of our network.
• R3 will be configured as the master controller but also as a border router. You can configure both on the same router.
• R3 has two serial links that we will use as “WAN” links. The serial 1/1 interface has a bandwidth of 64 kbps and the serial 1/0 interface has a bandwidth of 1024 kbps.
• R1 and R2 will be used as “traffic generators”.
• R1 will have a TCP connection to the loopback0 interface of R6.
• R2 will send ICMP traffic to the loopback1 interface of R6.
• R4 and R5 are nothing special, they are only used as endpoint for the serial links.
• R6 is only used as an endpoint for our “traffic generators” R1 and R2.

By default I will send all traffic over the slow 64 kbps link using a static route, we will use a floating static route to use the 1024 kbps link as a backup. The goal of this lab example is to configure OER to automatically switch traffic flows from the slow 64 kbps link to the 1024 kbps link. Sounds like fun right?

Let’s get to the configuration part!

To keep things simple I will use static routes for connectivity:

R1(config)#ip route 0.0.0.0 0.0.0.0 192.168.123.3

R2(config)#ip route 0.0.0.0 0.0.0.0 192.168.123.3

R1 and R2 will use a static route pointing to R3.

R3(config)#ip route 0.0.0.0 0.0.0.0 192.168.35.5
R3(config)#ip route 0.0.0.0 0.0.0.0 192.168.34.4 5

R3 has two default routes. The first one points to R5 and this is the static route that we will find in the routing table. The second static route is our floating static route pointing to R4. It has an administrative distance of 5.

Something that you should remember about OER is that it requires a “parent route” in order to use a certain link. If I didn’t configure the static route pointing to R4 than OER will never be able to use the serial 1/0 interface!

R4(config)#ip route 192.168.123.0 255.255.255.0 192.168.34.3
R4(config)#ip route 6.6.6.0 255.255.255.0 192.168.46.6
R4(config)#ip route 66.66.66.0 255.255.255.0 192.168.46.6

R5(config)#ip route 192.168.123.0 255.255.255.0 192.168.35.3
R5(config)#ip route 6.6.6.0 255.255.255.0 192.168.56.6
R5(config)#ip route 66.66.66.0 255.255.255.0 192.168.56.6

On R4 and R5 we will configure static routes so they can reach the 192.168.123.0 /24 network and the loopback interfaces of R6.

R6(config)#ip route 0.0.0.0 0.0.0.0 192.168.46.4
R6(config)#ip route 0.0.0.0 0.0.0.0 192.168.56.5

On R6 we’ll configure two default routes pointing towards R4 and R5. I don’t care which path R6 will use…

Let’s check if R1 and R2 can reach R6:

R1#ping 6.6.6.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/11/12 ms

R1#ping 66.66.66.66

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 66.66.66.66, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/16/32 ms

R2#ping 6.6.6.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/11/12 ms

R2#ping 66.66.66.66

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 66.66.66.66, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/16/32 ms

R1 and R2 can reach R6 so my static routes are working. Let’s check the path that they are currently using:

R1#traceroute 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

1 192.168.123.3 4 msec 4 msec 4 msec
2 192.168.35.5 8 msec 8 msec 8 msec
3 192.168.56.6 8 msec *  8 msec

We are using the path from R3 to R5, this is because of the default route that I configured on R3:

R3#show ip route static
S*   0.0.0.0/0 [1/0] via 192.168.35.5

So far so good, connectivity is working. Before I start with the configuration of OER I want to change the bandwidth of the serial interfaces on R3:

R3(config)#interface serial 1/1
R3(config-if)#bandwidth 64
R3(config-if)#exit
R3(config)#interface serial 1/0
R3(config-if)#bandwidth 1024
R3(config-if)#exit

R3#show interfaces serial 1/1 | include BW
MTU 1500 bytes, BW 64 Kbit, DLY 20000 usec,

R3#show interfaces serial 1/0 | include BW
MTU 1500 bytes, BW 1024 Kbit, DLY 20000 usec,

R3 now thinks that serial 1/1 is 64kbit and serial 1/0 is 1024 kbit. Of course this doesn’t change the actual bandwidth but OER will believe that this is the bandwidth of the interfaces. Now let’s take a look at OER!

When we configure OER we have to configure the master controller and the border router(s). OER needs to know what the internal and external interfaces are. R3 will be both a border router and the master controller. The FastEthernet 0/0 interface is internal and the two serial links are external. OER requires authentication and it has to be done with a key chain just like EIGRP. Here’s what the configuration looks like:

R3(config)#key chain OER
R3(config-keychain)#key 1
R3(config-keychain-key)#key-string NETWORKLESSONS

I’ll keep it simple, the key chain is called “OER” and the password will be “NETWORKLESSONS”. Now let’s configure the master controller role:

R3(config)#oer master
R3(config-oer-mc)#border 192.168.123.3 key-chain OER
R3(config-oer-mc-br)#interface fastethernet 0/0 internal
R3(config-oer-mc-br)#interface serial1/1 external
R3(config-oer-mc-br-if)#exit     
R3(config-oer-mc-br)#interface serial1/0 external
R3(config-oer-mc-br-if)#exit
R3(config-oer-mc-br)#exit
R3(config-oer-mc)#exit

First I use the oer master command to enable the MC role. Next step is to configure the border router and specify the key-chain. In my example I’m using IP address 192.168.123.3 but on a production network it’s better to use loopback interfaces to establish the neighbor adjacency between the master controller and border router(s). OER uses a TCP connection in case you were wondering.

I also configured the interfaces and specified whether they are internal or external, this is something you have to do for all border routers in the network! Now let’s configure the border router role:

R3(config)#oer border
R3(config-oer-br)#local fastEthernet 0/0
R3(config-oer-br)#master 192.168.123.3 key-chain OER

Use the oer border command to get into the border router configuration. The local command specifies the interface that will be used to source the TCP connection from. Last but not least we configure the master controller and the key chain. Now before we continue it’s a good idea to check if the border router has successfully connected to the master controller:

R3#show oer border
OER BR 192.168.123.3 ACTIVE, MC 192.168.123.3 UP/DOWN: UP 00:01:24,
Auth Failures: 0
Conn Status: SUCCESS, PORT: 3949
Exits
Fa0/0 INTERNAL
Se1/0 EXTERNAL
Se1/1 EXTERNAL

Use show oer border to verify your configuration. We can see that it has successfully connected to the master controller and we can see that it uses TCP port 3949. We also see the internal and external interfaces.

We can also take a look at the master controller part, here’s how to do it:

R3#show oer master
OER state: ENABLED and ACTIVE
  Conn Status: SUCCESS, PORT: 3949
  Number of Border routers: 1
  Number of Exits: 2
  Number of monitored prefixes: 0 (max 5000)
  Max prefixes: total 5000 learn 2500
  Prefix count: total 0, learn 0, cfg 0

Border           Status   UP/DOWN             AuthFail
192.168.123.3    ACTIVE   UP       00:03:31          0

Global Settings:
  max-range-utilization percent 20
  mode route metric bgp local-pref 5000
  mode route metric static tag 5000
  trace probe delay 1000
  no logging

Default Policy Settings:
  backoff 300 3000 300
  delay relative 50
  holddown 300
  periodic 0
  mode route observe 
  mode monitor both
  mode select-exit good
  loss relative 10
  unreachable relative 50
  resolve delay priority 11 variance 20
  resolve utilization priority 12 variance 20

Learn Settings:
  current state : DISABLED
  time remaining in current state : 0 seconds
  no throughput
  no delay
  no protocol
  monitor-period 5
  periodic-interval 120
  aggregation-type prefix-length 24
  prefixes 100
  expire after time 720

Use show oer master to see information from the MC’s role. We see that the master controller is running, we have a border router and two exit paths for our network. It also shows us the default policy. Explaining the policy settings it outside the scope of this lesson as there’s quite some stuff we can do. I will cover it later in another lesson.

Before I start traffic generation on R1 and R2 I will enable logging on the master controller. This wil output everything on the console, it’s just like a debug and it will show us what OER is doing:

R3(config)#oer master
R3(config-oer-mc)#logging

Now let’s generate some traffic from R1 and R2, here’s what I will do:

• R1 will connect to IP address 6.6.6.6 using TCP.
• R2 will send ICMP traffic to 66.66.66.66.

R6(config)#service tcp-small-servers

R1#telnet 6.6.6.6 19
Trying 6.6.6.6, 19 ... Open
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefg
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefgh
"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghi
#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghij
$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijk

I will enable TCP small servers on R6 and connect to TCP port 19 from R1. This will generate characters and it’s a great way to have a working TCP connection.

R2#ping 66.66.66.66 repeat 999999999 size 1500

On R2 we will configure a simple ping with a size of 1500. Make sure you send enough pings…

OER will enable netflow on the border routers and we can take a look at the current flows:

R3#show ip cache flow 
IP packet size distribution (17715 total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .000 .012 .000 .576 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .000 .000 .000 .000 .410 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
  4 active, 4092 inactive, 7 added
  589 ager polls, 0 flow alloc failures
  Active flows timeout in 30 minutes
  Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 21640 bytes
  7 active, 1017 inactive, 11 added, 7 added to flow
  0 alloc failures, 0 force free
  1 chunk, 1 chunk added
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-other            1      0.0         1    44      0.0       0.0      15.9
ICMP                 2      0.0        85  1500      0.0       1.1      15.1
Total:               3      0.0        57  1491      0.0       0.7      15.3

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Se1/1         6.6.6.6         Fa0/0         192.168.123.1   06 0013 B154    10K
Fa0/0         192.168.123.1   Se1/1         6.6.6.6         06 B154 0013   226 
Fa0/0         192.168.123.2   Se1/1         66.66.66.66     01 0000 0800  3610 
Se1/1         66.66.66.66     Fa0/0         192.168.123.2   01 0000 0000  3610 

Above you see my ICMP and TCP flow. You’ll see it from both directions and you can see the number of packets/bytes etc. There is a useful command on the master controller that we can use to see the current load on our interfaces:

R3#show oer master border detail 
Border           Status   UP/DOWN             AuthFail
192.168.123.3    ACTIVE   UP       00:19:32          0
 Se1/0           EXTERNAL UP             
 Se1/1           EXTERNAL UP             
 Fa0/0           INTERNAL UP             

 External         Capacity      Max BW   BW Used Tx Load Status          Exit Id
 Interface         (kbps)       (kbps)    (kbps)    (%)                         
 ---------        --------      ------   ------- ------- ------           ------
 Se1/0                1024         768         0       0 UP                    2
 Se1/1                  64          48       631      67 UP                    1

Use the show oer master border detail command to see the current traffic load on the interfaces of your border routers. Above you can see the current traffic load on the serial 1/1 interface which is 631kbps. You can see the capacity of the links and the max BW. The maximum bandwidth that we can use is 75% of the interface. This serial 1/1 interface is heavily oversubscribed right now while our serial 1/0 interface is doing nothing. This is normal because our default route is pointing out the serial 1/1 interface. If you enabled logging for OER then you will also see messages like these in your console:

R3#
%OER_MC-5-NOTICE: Range OOP BR 192.168.123.3, i/f Se1/1, percent 71
%OER_MC-5-NOTICE: Load OOP BR 192.168.123.3, i/f Se1/1, load 865 policy 48
%OER_MC-5-NOTICE: Exit 192.168.123.3 intf Se1/1 OOP, Tx BW 865, Rx BW 909, Tx Load 94, Rx Load 94

We haven’t talked about policies but OOP means “Out of Policy“. The default policy of OER is telling us that there’s too much traffic on our serial 1/1 interface right now.

OER is giving us all this great information but at the moment it’s not taking any action. First we need to configure it to learn about the prefixes that have the highest throughput or delay. OER uses the information from netflow for this, here’s how we can enable the learning of those prefixes:

R3(config)#oer master 
R3(config-oer-mc)#learn 
R3(config-oer-mc-learn)#throughput
R3(config-oer-mc-learn)#delay

Throughput means it will learn prefixes with the highest outbound throughput, delay means prefixes with the highest RTT (Round Trip Time). As soon as you enable this you will see the following in your console:

R3(config-oer-mc)#
%OER_MC-5-NOTICE: Prefix Learning STARTED

By default OER will learn prefixes for 5 minutes and then sleep for 120 minutes. This can be annoying in a lab so you might want to speed it up. If you miss the 5 minute “learning” period you’ll have to wait 120 minutes (or reset OER). These timers can be tuned if you want. Once the learning period is over (5 minutes) you will see this in the console:

R3(config-oer-mc)#
%OER_MC-5-NOTICE: Prefix Learning WRITING DATA

OER will have learned about the top prefixes but it will still not take any action.This is because the default mode is to observe but not take any action.  Information from the border routers is sent to the master controller but that’s it. You can verify the current OER mode here:

R3#show oer master | include mode route
  mode route metric bgp local-pref 5000
  mode route metric static tag 5000
  mode route observe 

Now we will change the mode to control so that the master controller will send instructions to the border router! Here’s how to do it:

R3(config)#oer master
R3(config-oer-mc)#mode route control 

Use the mode route control to make OER send instructions back to the border routers. The master controller will now instruct our border router to change the routes, let’s see if this is true:

R3#show oer master border detail 
Border           Status   UP/DOWN             AuthFail
192.168.123.3    ACTIVE   UP       01:10:09          0
 Se1/0           EXTERNAL UP             
 Se1/1           EXTERNAL UP             
 Fa0/0           INTERNAL UP             

 External         Capacity      Max BW   BW Used Tx Load Status          Exit Id
 Interface         (kbps)       (kbps)    (kbps)    (%)                         
 ---------        --------      ------   ------- ------- ------           ------
 Se1/0                1024         768       222      21 UP                    2
 Se1/1                  64          48       603      38 UP                    1

Awesome! As you can see it is now also using the serial1/0 interface. Let’s take a look what OER did exactly:

R3#show oer border routes static 

Flags: C - Controlled by oer, X - Path is excluded from control, 
       E - The control is exact, N - The control is non-exact

Flags Network            Parent             Tag       
CE    6.6.6.0/24         0.0.0.0/0          5000      
CE    66.66.66.0/24      0.0.0.0/0          5000   

Use show oer border static to see what routes OER has changed. It has influence traffic towards 6.6.6.0 /24 and 66.66.66.0 /24. Let’s take a look at the routing table:

R3#show ip route static 
     66.0.0.0/24 is subnetted, 1 subnets
S       66.66.66.0 [1/0] via 192.168.34.4
     6.0.0.0/24 is subnetted, 1 subnets
S       6.6.6.0 [1/0] via 192.168.34.4
S*   0.0.0.0/0 [1/0] via 192.168.35.5

Do you see the next hop IP address of the two static routes that I highlighted? Traffic for these two networks is now sent towards R4. OER has configured this for us!…pretty cool right?

This concludes our basic OER configuration tutorial. You have now learned how to configure a master controller, border controller, how to make OER learn about prefixes from netflow and how to take action upon it. In future lesson I will explain more about the different policies and how to use OER in combination with BGP. If you enjoyed this lesson please leave a comment and feel free to ask any questions!