One of the fundamental aspects of communication is the amount of information that can be communicated in one chunk before allowing others to communicate through the same channel; the maximum transmission unit (MTU).

In computer communications the MTU is the amount of data transportable across the lowest layer of the communication stack; Ethernet, FDDI, etc. From the base MTU each layer above in the networking stack breaks the data that it has to send into packets that will fit in the next layer down. For TCP/IP over Ethernet the base Ethernet specification includes a MTU of 1500 bytes (octets in the IETF RFC) not including the Ethernet header and the TCP/IP layers expand their default packet size from 536 and 576 bytes respectively to 1460 and 1500. Effectively the MTU sets the packet size of higher protocols in the OSI model.

Networks are often unable to transport a maximum sized packet. In which case the packets are fragmented either by the sender and receiver or some device(s) in the network in between. There are many reasons for this, for example:

  • Change in base network media i.e. Ethernet to FDDI conversion.
  • Poor line quality.
  • Protocol tunnelling consuming part of the transmission with embedded headers etc.
  • Network device capacity.
  • Data transmission streams with conflicting efficient packet size efficiencies i.e. FTP and VOIP.

Using the standard MTU dynamic shaping protocols normally gets around the these transmission problems. If a device can’t handle a delivered MTU it responds with an ICMP message that informs the sender and advertises its acceptable MTU. However it doesn’t always work and can produce various effects from black hole routing, to delayed responses and network application transmission jitter (very annoying in media streaming applications). Some of the reasons for this are hardend or unsophisticated systems that do not support dynamic MTU discovery or IPsec bridges or other tunnels improperly configured.

To remedy issues it is best to replace the networking infrastructure that is causing the issue with components that are well behaved and can support dynamic MTU. Some times however this is not possible so you have two options, configure a gateway to the device to clamp the MTU to an acceptable size (which you have already found out) or manually configure the sending devices to have an appropriate MTU for the affected route. If you have Windows devices then your best option is to use an existing or introduce a new Unix/Linux based gateway to clamp for you.

Linux (specifically Redhat/Fedora distributions) provides many facilities for the manipulation of MTU. The tools include the ‘/proc’ system components and various applications from the nettools package as well as our old friend ifconfig. So the facilities that I generally use at the start are:

  • ifconfig – to view the MTU and other network information.
  • tcpdump – to look for MTU dynamic shaping ICMP responses.
  • ethtool – to set the MTU and other network card options.
  • route – for fixed MTU setting based on route.
  • ‘/proc/sys/net/ipv4/ip_no_pmtu_disc’ – for fixing the MTU for testing.

ifconfig is the first place to start, it will tell you without any fuss what your network card is set to. A simple ifconfig on the command line will give you something that looks like this:

[root@bluetop ipv4]# ifconfig
 eth0      Link encap:Ethernet  HWaddr 00:0B:DB:19:46:4D
 inet addr:10.0.0.176  Bcast:10.0.0.255  Mask:255.255.255.0
 UP BROADCAST MULTICAST  MTU:1500  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
 Interrupt:10
lo        Link encap:Local Loopback
 inet addr:127.0.0.1  Mask:255.0.0.0
 inet6 addr: ::1/128 Scope:Host
 UP LOOPBACK RUNNING  MTU:16436  Metric:1
 RX packets:6178 errors:0 dropped:0 overruns:0 frame:0
 TX packets:6178 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:309060 (301.8 KiB)  TX bytes:309060 (301.8 KiB)
wlan0     Link encap:Ethernet  HWaddr 00:11:50:FD:7B:65
 inet addr:192.168.2.98  Bcast:192.168.2.255  Mask:255.255.255.0
 inet6 addr: fe80::211:50ff:fefd:7b65/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 RX packets:4126 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2447 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:4505043 (4.2 MiB)  TX bytes:393742 (384.5 KiB)

From this you can see that I’ve three interfaces running and eth0 is not being used. However note that the MTU value of each interface is shown here. This is the MTU for the device but not the path. Note also that the MTU for a particular path may be much smaller. In Fedora this value can be set a number of ways but the two simplest are the GUI network tool, which is very simple just look at it as it doesn’t need to be discussed here discussed here, and once again ifconfig:

[root@bluetop ipv4]# ifconfig lo
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16430  Metric:1
          RX packets:6417 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6417 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:336104 (328.2 KiB)  TX bytes:336104 (328.2 KiB)
[root@bluetop ipv4]# ifconfig lo mtu 16436
[root@bluetop ipv4]# ifconfig lo
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:6417 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6417 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:336104 (328.2 KiB)  TX bytes:336104 (328.2 KiB)

In the example above the loopback device has had its MTU set to 16436 from 16430. If we echo 1 to ‘/proc/sys/net/ipv4/ip_no_pmtu_disc’ then MTU dynamic shaping will be turned off and the machine will not send data packets less than the maximum if it has data to fill it. This is some times useful when trying to simulate devices on a network that do not have dynamic MTU built in or disabled.

ip is another tool that can be used for manipulating the MTU. But instead of altering the MTU for the whole device it can be targeted to a specific network. This is especially useful when a network device is shared between different networks. This tool can do all sorts of things so its a good place to start when considering things like tunnelling and complex routing. But here we’ll stick to setting MTU’s on specific routes. For example lets say that we’ve discovered that the network 10.0.0.0 really needs to have an MTU of 1400 but its behind our default gateway and its just not going to work out well for all of the other networks if we clamp to an MTU of 1400 for everyone. So here’s the answer:

[root@bluetop ipv4]# route -ee
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface    MSS   Window irtt
192.168.2.0     *               255.255.255.0   U     0      0        0 wlan0    0     0      0
default         192.168.2.1     0.0.0.0         UG    0      0        0 wlan0    0     0      0
[root@bluetop ipv4]# ip route add 10.0.0.0/24 via 192.168.2.1 mtu 1400
[root@bluetop ipv4]# ip route show
10.0.0.0/24 via 192.168.2.1 dev wlan0  mtu 1400
192.168.2.0/24 dev wlan0  proto kernel  scope link  src 192.168.2.98
default via 192.168.2.1 dev wlan0  proto static
[root@bluetop ipv4]# route -ee
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface    MSS   Window irtt
10.0.0.0        192.168.2.1     255.255.255.0   UG    0      0        0 wlan0    0     0      0
192.168.2.0     *               255.255.255.0   U     0      0        0 wlan0    0     0      0
default         192.168.2.1     0.0.0.0         UG    0      0        0 wlan0    0     0      0

Neat? Well this can be surprisingly useful in many places. In fact the inverse of the previous example is uncommonly useful when dealing with an artificially clamped default path. Consider your ADSL connection which probably uses PPoE (Point to Point over Ethernet) tunnelling. Your local machine undoubtedly uses an Ethernet MTU of 1500 and the ADSL device probably does on the connection to your network provider (which is invariably different from your ISP). However PPoE consumes a little of that 1500 bytes to pack in its tunnelling protocol. This means that on large data transfers your machine is pumping out 1500 bytes of TCP/IP to your ADSL connection and that device is quite likely fragmenting those packets into 2 so that it can get its tunnelling protocol messages into the same packets. Effectively you are adding a small but significant overhead to the whole data transfer process increasing the consumption of your ISP data cap and causing your ADSL device to work harder than required. Yes you are in fact solely responsible for global warming. You crim!

Often in professional networks where PPoE tunnels are used you will see various counter measures to avoid the side effects of the tunnelling overhead. One way is to configure jumbo (oversized MTU) packets on the tunnelling devices. However that really requires you to configure both ends and its not really normal for you to be able to configure your network supplier’s routers. At least not legally. However you can clamp your default routes from your devices and set your local LAN MTU to, well, whatever you want. That way you don’t affect your local LAN performance and you can improve your gateway throughput.

Naturally this is reasonably dangerous so I’m just going to give you the command and let you break your network yourself. I guarantee that you will stuff this up so have a play when you don’t mind spending the time on it; and by the way this sort of stupidity can raise the temperature on your physical devices: so you can actually physically damage your equipment or alter its life expectancy this way. BE WARNED you own ALL of the risk on this one. But do have fun!

ip route add default via 10.0.0.1 mtu 1470