Go to:

IT Jobs

Most network administrators are familiar with freely available network diagnostic tools such as Wireshark and TCPdump. However, many may not realise that the Internet2 consortium has also produced several advanced open-source tools that, while designed to monitor and troubleshoot performance issues on high-performance research networks, can be great additions to any networker's bag of tricks.

In particular, the Internet2 consortium has developed a software package for the End To End Performance Initiative called Performance Improvement Performance Environment System (PIPES). One particularly useful component of PIPES is the Network Diagnostic Tester (NDT).

NDT was developed by Richard Carlson, an engineer at Internet2. Carlson developed a signature-based engine that forms the core of NDT and the integration to make the system work. The result is a reliable, stable tool that differentiates itself from other common tools in greatly assisting network administrators in diagnosing network problems.

I have often referred my customers to throughput testing sites such as McAfee's Internet Speedometer or DSLreports.com to test their bandwidth. However, these sites are only as accurate as the slowest pipe between client and server, and this almost always means the Internet connection. Their usefulness breaks down when measuring throughput within a local-area network.

A useful method for measuring throughput on a LAN is to use File Transfer Protocol to transfer a large file. FTP reports the average throughput for the transfer. However, FTP cannot report if network congestion or incorrectly configured links are adversely affecting the results.

NDT handles these issues, and much more. It's simple to set up on a Linux server with a web100-patched kernel. More importantly, the web interface is simple enough for an end user to use, requiring only a click of the "Start" button and cutting and pasting the results in an e-mail to send to the networking staff.

NDT is designed to work well as a component of a LAN support structure. Carlson says he envisions NDT "being integrated into the help desk so when a trouble call is received, the first thing the NOC staff does is point the user to their local NDT server and say, 'Run a test and send us the results.'"

Using NDT
While NDT is designed to be installed on a server, I realised the potential of installing it on a laptop to allow for mobility. With this configuration, an NDT-enabled laptop can easily be inserted on the network at locations to maximise testing. In other words, by eliminating the location constraints of the NDT server, I could test throughput without the added variables of router hops and additional intermediate switches.

The following examples illustrate cases where NDT was an indispensable aid for determining the cause of network performance problems.

Case 1: The LAN
I was asked to diagnose network connectivity problems within a LAN, specifically from a client to a server on the same subnet but not on the same switch. The user was convinced that it was a bandwidth congestion issue. However, I suspected a physical problem or a link issue, as switch statistics did not indicate traffic patterns deviating from baseline measurements.

I first ran a ping test to the switch the server was on from the client's machine. The result was a loss of about 5 percent of the packets. However, the switches employed on this particular network had the "feature" of assigning a low priority to Internet Control Messaging Protocol echo packets. It was not uncommon for a ping test to this type of switch in a stand-alone configuration to drop a low percentage of packets. Therefore, this test produced inconclusive results, and I realised I had to dig deeper.

I placed the mobile NDT server on the same switch that the client's server was attached to and went to the client's machine to run an NDT test. NDT reported a mismatched duplex in the path between the client and server. Finding the mismatch and correcting it resolved the issue.

Case 2: The ISP
I was working on transitioning a site's Internet link from a DS3 to a Gigabit Metro Ethernet throttled to 100Mbit/s. My preparation for the cutover included placing the mobile NDT server on a switch where the Metro Ethernet terminated.

This was a unique case, in that the Metro Ethernet terminated in the same remote Internet service provider router as the DS3. Since both links were operational concurrently before the cutover, I was able to perform a test from the client site, out the DS3 and in the Metro Ethernet.

An NDT test described above showed a throughput of about 5Mbit/s and correctly identified the slowest link as a DS3. I expected a greater throughput because the DS3 was carrying very little production traffic at this time. Pings across the links showed no loss.

Since all I had was an uneasy feeling about the NDT throughput results being lower than I expected, I reluctantly agreed to a service cut. I assumed the lower throughput value reported by NDT was an anomaly not related to the circuits, but I prepared for a rapid return to the DS3 just in case.

After the cut, with the Metro Ethernet now under a nearly negligible traffic load of approximately 4 percent inbound, I ran a ping test that revealed a packet loss of about 3 percent. While the Internet service provider pronounced the circuit healthy, my ping results coupled with the low throughput as determined by NDT were enough for me to make the call to reverse the cut.

As it turned out, there was a negotiation settings mismatch between two Internet service provider devices providing the Metro Ethernet link. Gigabit Ethernet works very well with autonegotiation turned on, but when one side is set for such and the other is not, odd errors can occur.

In this example, if I had not performed an NDT bandwidth test prior to cutover, I may have agreed with the service provider that the circuit was good to go. This would have resulted in performance problems the next business day when the circuit was under operational loads. Realising the problem prior to this was indispensable.

The future
Carlson reports that some planned enhances to NDT include IPv6 support and some firewall testing features. In addition, the NDT as well as other components of the PIPES suite are now available as an ISO image, thereby reducing the download and installation steps.

While I consider NDT to be an indispensable tool for the network administrator, I have found other components of PIPES very useful as well. Development is ongoing, and each tool can provide measurement statistics previously unavailable. In today's complex network environments of interoperability, filtering, firewalling and bandwidth management, these tools can aid in pointing out issues missed by the traditional methods.