MTU Troubleshooting on Cisco IOS

Maximum Transmission Unit (MTU) is the largest size in bytes that a certain layer can forward. The MTU is different for each protocol and medium that we use. Ethernet for example has a MTU of 1500 bytes by default.

This means that a single Ethernet frame can carry up to 1500 bytes of data. On top of this data we add the Ethernet header. Typical header sizes are 14 bytes for Ethernet (add another 4 bytes if you use 802.1Q tagging).

Below is a picture with the different MTU fields that you might encounter:

You can see that a typical Ethernet header is 14 bytes, IP is 20 bytes and TCP is also 20 bytes. The maximum amount of payload that TCP can use is called the TCP MSS (Maximum Segment Size). This MSS value is the largest amount of data that a host can receive in a single TCP segment. This value is used to set a limit on the payload in order to prevent fragmentation and is sent in the SYN packet during the 3 way handshake. The MSS value isn’t synchronized between hosts, it can be different for each direction.

So why is all of this important to know? Let’s imagine we have an IP packet that is sent on our LAN. The size of the Ethernet frame will be like this:

• 1460 bytes of payload for TCP.
• 20 bytes for the TCP header.
• 20 bytes for the IP header.
• 14 bytes for the Ethernet header.

1460 (PAYLOAD) + 20 (TCP)  + 20 (IP) = 1500 bytes + 14 (ETHERNET) = 1514 bytes in total.

Sending 1514 bytes is no problem for Ethernet but some other mediums might have issues with large MTU values. Often problems arise when you add additional headers because of PPPoE, GRE Tunneling or IPSEC since this reduces the available bytes for our MTU. To demonstrate this problem (and how to solve it!) I will use a simple network with a reduced MTU. Here’s what it looks like:

The network above has two routers, a webserver (S1) behind R1 and a client (H1) behind R2. Here’s what the configuration looks like:

Configuration

First we’ll configure some IP addresses:

R1(config)#interface fastEthernet 0/0
R1(config-if)#ip address 192.168.12.1 255.255.255.0

R1(config)#interface fastEthernet 0/1           
R1(config-if)#ip address 192.168.1.254 255.255.255.0

R2(config)#interface fastEthernet 0/0
R2(config-if)#ip address 192.168.12.2 255.255.255.0

R2(config)#interface fastEthernet 0/1
R2(config-if)#ip address 192.168.2.254 255.255.255.0

I’ll add some static routes for connectivity:

R1(config)#ip route 192.168.2.0 255.255.255.0 192.168.12.2

R2(config)#ip route 192.168.1.0 255.255.255.0 192.168.12.1

Here’s what the default MTU values look like:

R2#show interfaces fastEthernet 0/0 | include MTU
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, 

R2#show ip interface fastEthernet 0/0 | include MTU
  MTU is 1500 bytes

The first MTU value is the interface MTU, it’s 1500 bytes by default for Ethernet. The second one is the IP MTU which is also 1500 bytes. Once you get above 1500 bytes your router will start fragmenting the IP packets.

Is this limit of 1500 bytes really working? There’s an easy way to find out. Let’s do a ping with the DF-bit (Don’t Fragment) between the routers:

R2#ping
Protocol [ip]:   
Target IP address: 192.168.12.1
Repeat count [5]: 1
Datagram size [100]: 
Timeout in seconds [2]: 
Extended commands [n]: y
Source address or interface: 
Type of service [0]: 
Set DF bit in IP header? [no]: y
Validate reply data? [no]: 
Data pattern [0xABCD]: 
Loose, Strict, Record, Timestamp, Verbose[none]: v
Loose, Strict, Record, Timestamp, Verbose[V]: 
Sweep range of sizes [n]: y
Sweep min size [36]: 1495
Sweep max size [18024]: 1505
Sweep interval [1]: 
Type escape sequence to abort.
Sending 11, [1495..1505]-byte ICMP Echos to 192.168.12.1, timeout is 2 seconds:
Packet sent with the DF bit set
Reply to request 0 (1 ms) (size 1495)
Reply to request 1 (4 ms) (size 1496)
Reply to request 2 (1 ms) (size 1497)
Reply to request 3 (4 ms) (size 1498)
Reply to request 4 (1 ms) (size 1499)
Reply to request 5 (4 ms) (size 1500)
Request 6 timed out (size 1501)
Request 7 timed out (size 1502)
Request 8 timed out (size 1503)
Request 9 timed out (size 1504)
Request 10 timed out (size 1505)
Success rate is 54 percent (6/11), round-trip min/avg/max = 1/2/4 ms

In the ping above you can see that the largest packet that I can send is 1500 bytes. The second packet with 1501 bytes can’t be sent because it is too large and we set the DF-bit.

Let’s look at an actual packet between the client and the webserver, see what these values look like in an actual frame:

Above you can see the TCP MSS which is 1460 bytes. What else can we see in Wireshark?

Above you see that the total length of the IP packet is 1500 bytes (1460 bytes for TCP MSS + 40 bytes for the TCP/IP header). Ethernet adds another 14 bytes which is how we get to 1514 bytes in total.

To simulate a network with a lower MTU I will reduce the MTU of the FastEthernet 0/0 interface of R2:

R2(config)#interface fastEthernet 0/0
R2(config-if)#mtu 1400

By reducing the MTU to 1400 bytes, the largest TCP segment size will be only 1360 bytes (1400 – 40 = 1360).  This is a scenario where users often complain that sending a ping is no problem but accessing a website or something else doesn’t work. Why? Let’s look at the example below:

Above you see a ping between the client and webserver. As you can see the total length is only 74 bytes…no problem to send this because our MTU allows 1400 bytes. Now we will try to connect to the webserver from the client using HTTP:

This is where things go wrong…as you can see above the total length of the IP Packet is 1500 bytes which is 100 bytes above our MTU of 1400 bytes. It’s impossible to communicate between the client and webserver now.

How do we fix this? There are two solutions for this:

• Set the correct IP MTU value so the router knows when to fragment IP packets.
• Reduce the TCP MSS value for outgoing connections so there is less payload.

Here’s how to configure the correct IP MTU value:

R2(config)#interface fastEthernet 0/0
R2(config-if)#ip mtu 1400

This tells the router to fragment all outgoing IP packets when they exceed 1400 bytes. The second step is to tell the router to intercept all TCP SYN packets from hosts and change the MSS value. We do this on the interface that is facing our hosts:

R2(config)#interface fastEthernet 0/1
R2(config-if)#ip tcp adjust-mss 1360

Setting the TCP MSS value to 1360 ensures that our total MTU will be 1400 bytes (1360 + 40). Does this work? Let’s look at a Wireshark example between our host and webserver again:

Above you can see that the maximum segment size is now 1360 bytes. What about the total packet length? Look below…

With a TCP MSS of 1360 our IP packet is only 1400 bytes which is exactly our MTU of 1400 bytes. You will notice that all communication problems between the host and webserver are now solved.

The IP tcp adjust-mss command is to intercept TCP SYN packets from hosts and set the correct MSS value, if you want to change the MSS for TCP connections that are originated by the router you need to use another command:

R2(config)#ip tcp mss 1360

The ip tcp mss command changes the MSS for the router itself. We can do a debug to check if it’s working or not:

R2#debug ip tcp transactions 
TCP special event debugging is on

Let’s use telnet to connect to TCP port 80 to the webserver:

R2#telnet 192.168.1.1 80
Trying 192.168.1.1, 80 ... Open

TCB495B8B94 created
TCB495B8B94 setting property TCP_VRFTABLEID (20) 496CF85C
TCB495B8B94 setting property TCP_TOS (11) 496CF7F8
TCB495B8B94 setting property TCP_RTRANSTMO (31) 496CF728
TCB495B8B94 setting property TCP_GIVEUP (34) 496CF72C
TCP: Random local port generated 36352, network 1
TCB495B8B94 bound to UNKNOWN.36352
TCB495B8B94 setting property unknown (24) 496CF758
Reserved port 36352 in Transport Port Agent for TCP IP type 1
TCP: sending SYN, seq 3882550463, ack 0
TCP0: Connection to 192.168.1.1:80, advertising MSS 1360
TCP0: state was CLOSED -> SYNSENT [36352 -> 192.168.1.1(80)]
TCP0: state was SYNSENT -> ESTAB [36352 -> 192.168.1.1(80)]
TCP: tcb 495B8B94 connection to 192.168.1.1:80, peer MSS 1460, MSS is 1360

You can see that R2 is using a MSS value of 1360 now…problem solved.

Hopefully this tutorial is useful for you to solve some of the MTU problems! If you have any questions, feel free to leave a comment.