This page contains detailed technical information to help you get the best speed and performance out of your KAREN connection.

Network speed

Potential bottlenecks

TCP tuning

Jumbo MTU

Routing management

Network speed 

The speed of the core network has been thoroughly tested and provides predictable performance, even under extreme loading.

Domestic latency has been measured at 22 milliseconds (ms) round trip time (RTT) between the two most distant POPs on the network - North Shore POP and Invermay POP. The jitter measurements have been less than 40 microseconds (ųsecs) or 0.04 ms.

The network comprises dual 10Gbps paths that provide resiliency and can be load balanced to provide 20Gbps when both paths are available.

While the network comprises significant bandwidth with low latency there are a variety of issues that will limit the effective use that can be made of the network.

Potential bottlenecks

Connections are offered to members at either 1 or 10 Gbps. The throughput that is achievable by the customer premise equipment (CPE) should at a minimum be equal to the speed of the connection that has been chosen.

While many devices have gigabit capable interfaces (often GBIC’s), some of them only support a fraction of that speed through the device and this fraction can be effected by service that are configured on the device.

This issue needs to be considered for every piece of your network infrastructure including the desktops or host devices themselves.

System bus/expansion card

A good example of this is the difference between PCI Express (PCIe) and the other system bus/expansion card interface formats such as PCI Extended (PCI-X) and PCI.

top

In practice it will be difficult to tune the rest of the network infrastructure and the operating system to the extent that the choice between PCI, PCI-X and PCIe becomes an issue. However, it does highlight the differences in performance that can be created through the choices that are made around server and desktop components.

Mass storage devices

Many R&E interactions and applications require large volumes of data to be transferred across the network. This data will most likely be read from a mass storage device such as a disk and will often also need to be written to disk on arrival.

Disk technology has been steadily increasing potential transfer speeds. A simple example can be seen in the progression of SCSI standard.

Other mass storage solutions will all have potential bottlenecks whether it is a simple IDE hard disk or a complex network attached storage (NAS) solution.  Many of these solutions can be tuned for higher performance and the appropriate vendors should be able to supply documentation about the steps required. 

Network interface cards (NIC)

The NIC used should be able to support at least 1Gbps and support a frame size that will allow a 9000 byte IP MTU to be supported.

The NIC can also cause bottlenecks because the buffers on the (NIC) can only hold a finite amount of data.  If the bus or other components do not allow the NIC to transfer data out at the rate it is being received the buffers will eventually become full and errors will start to occur.

Interrupt request conflicts

Computer systems utilise interrupt requests (IRQs) to signal the CPU that there is data to be processed from a peripheral device. Most enterprise systems will not have this problem as significant testing is undertaken to ensure that these conflicts do not occur, however changes in configuration (H/W or S/W) should be properly audited. It may be appropriate on desktop machines with after market components installed to check for IRQ conflicts.

Despite the development of plug and play technology in many operating systems conflicts do still occur. These conflicts can cause applications to slow down until there are free CPU cycles to devote to the required tasks again.  This is probably less likely in newer systems with multi-threaded CPU’s.

General resource shortages

All computer applications require memory and CPU cycles in order to perform properly. Shortages in these resources will cause performance issues regardless of whether the activity in question is a simple data transfer or a multimedia streaming application.

top

TCP tuning

The TCP and IP protocol were developed in the 1970’s as a means to provide addressing and control for a small group of networks. The problems that these protocols were built to address are not the same ones that we face today, where networks are larger, transport is more reliable and connection speeds far higher.

TCP/IP has evolved into the default communication protocol for local area networks and the Global Internet, yet the default settings have remained mostly unchanged.

In order to enable TCP to deal effectively with the increase of available bandwidth that KAREN provides and corresponding reductions in latency and error rates some of the default settings have to be changed. To maximise use of bandwidth and reduce overhead and delay attributed to error control, the following needs to be taken into account:

Maximum TCP buffer space

Most operating systems place a limit on the amount of physical memory that can be used by any single TCP connection. This can become a bottleneck on networks such as KAREN that support high volumes of data transfer.

Socket buffer sizes

Most operating systems also support separate send and receive buffer limits on a per connection basis. These buffer limits can usually be adjusted by users, applications or other mechanisms within the maximum TCP buffer space.

TCP large window extensions (RFC 1323)

RFC 1323 covers TCP extensions for high performance including the window scale option and time stamps which are required to support networks such as KAREN.

The window scale option has two purposes:

1. indicate that the TCP is prepared to do both send and receive window scaling
2. communicate a scale factor to be applied to its receive window.

The time stamp option is used for two distinct mechanisms: Round Trip Time Measurement (RTTM) and Protect Against Wrapped Sequences (PAWS).  Both of these mechanisms provide a way of preventing or dealing with spurious retransmissions.

TCP selective acknowledgement option (RFC 2018)

Normal TCP transmissions involve acknowledgement messages for each segment of transmitted data. This option allows for selective acknowledgements pertaining to missing segments only.

Path maximum transmission unit discovery (RFC 1191)

Path MTU Discovery (PMTU) utilises Internet Control Message Protocol (ICMP) and the local network connection MTU to find the largest possible packet size that can be used in end-to-end communication. This is very important when using KAREN because jumbo MTU size is utilised. This is covered in more detail below.

It is important to understand that because TCP works in an end-to-end fashion both ends of the communication need to be.

top

Jumbo MTU

Jumbo MTU on Gigabit Ethernet networks provide increased data throughput in addition to reducing the amount of resources (e.g. CPU) required from network infrastructure.  Research has shown a 50% increase in throughput and a 50% decrease in CPU load when comparing 1500 and 9000 byte MTUs. The increased data throughput is vital in utilising a high speed, low latency network. See 'What is jumbo frame support?' for more information on jumbo frames.

KAREN utilises a 9000 byte IP MTU which is the same as the IP MTU of Internet2 and other research and education networks. This 9000 byte IP MTU must be supported from end-to-end to be effective in increasing data throughput. 

Provided that path MTU discovery is properly supported the MTU of an end-to-end transmission will be the smallest IP MTU across the path taken between the two systems.  Figure 1 and Figure 2 (below) provide a simple visual of how this would work.

Figure 1: 1500 byte end-to-end communication

Figure 1 depicts a situation where the entire end-to-end transmission is constrained by the device that supports only 1500 bytes (standard Ethernet frame size).  Assuming path MTU discovery is properly supported the communication will occur using 1500 byte MTU. If Path MTU Discovery is not supported the communication will not occur.

Figure 2: 9000 byte end-to-end communication

Figure 2 depicts a jumbo capable path that will allow end-to-end transmission using a 9000 byte MTU.  It is important to note that every active device in the network path must support the MTU.

It should also be noted that while jumbo MTU is very effective on Gigabit Ethernet or 10 Gigabit Ethernet, it is not suitable for Fast Ethernet (100Mbps) or Ethernet (10Mbps). These results can include link flapping and degraded performance through head of line blocking or serialisation delays that may adversely effect time critical communication like VOIP.

top

Routing management

KAREN peering uses BGP to provide routing information for network prefixes that are available through KAREN.  In many cases BGP will also be used to peer with other organisations or an internet service provider (ISP), which can create a routing table with multiple paths to the same destination network prefixes.

While this provides organisations with the flexibility to be able to make use of diverse network routes and control the outbound path that traffic takes, it can also create asymmetric routing or lead to the sub optimal routing of packets.

Sub optimal routing

It is important to understand the path your traffic will take to reach any particular destination. A clearly defined routing policy that is monitored and regularly analysed is the best defense against inefficient routing occurring. It is important to understand the path you believe traffic should take and compare that to the actual results where performance issues are encountered.

Traffic that is routed over the commodity internet or slower circuits instead of high speed, high bandwidth circuits will inevitably adversely affect performance.

Asymmetric routing

Asymmetric routing occurs when outbound traffic is sent on one path and return traffic takes a different path. Both routing paths are valid, and in the event of the primary path being unavailable the backup path would be used to reach the destination prefix.

However, when one path is significantly faster than the other path significant performance problems can be created when packets are sent using both paths. TCP/IP congestion avoidance mechanisms can also lead to the problem as the protocol stack slows down transmission believing the network to be congested, when the problem is potentially the use of an incorrect path.

This problem is significant for KAREN users because latency and jitter are significantly smaller and bandwidth significantly larger for KAREN routes than other routes.

Asymmetric routing can be difficult to discover as utilities such as traceroute may not provide evidence of the problem, unless the traceroute is performed separately in both directions. Once discovered there are a number of ways the problem can be addressed:

• Use routing policy to tune BGP so that the preferred route is predictable and not based on metrics that may change
• Work with the peer organisation that you are collaborating with to ensure you are both clear on the intended outcome.  Their BGP policy will impact the communication as much as yours will
• Consider advertising a smaller prefix into KAREN i.e. if you have a /16 at your disposal advertise that through the commodity Internet and a /24 through KAREN
• Make sure that all routes visible though KAREN are preferred. NREN’s generally operate the policy of preferring routes from other NREN’s so this should lead to consistent routing in both directions

Related resources

• Technical resources - for presentations on performance tuning
  

top

Updated 18 May 2009