Troubleshooting

Technical Training and Troubleshooting:

Protocol Analysis, Data Recorder, Application Performance, User Experience, Industrial Ethernet, Data Loss Prevention, Deep Packet Inspection, NetFlow, SOX, , Switching and Routing, Forensics Analysis, VoIP, IPTV

Author Profile – Ray Tompkins is the Founder and CEO of Gearbit. Ray is a Senior Network Specialist with over 32 years experience in troubleshooting, design, and implementation. His background includes 911 emergency consulting, and identifying the root cause of critical network problems. His knowledge of network protocols (LAN, VoIP, WAN and WLAN) and how they work within the enterprise networks are the key in providing customer service through knowledge transfer and education.

Under the Hood: Capture Filters, Display Filters

Figure 1:1 Wireshark: How to select Display Filters
 F1 

 

This month we will discuss the importance of filtering.

When conducting packet analysis, one of the most important tricks of the trade is filtering. The ability to either filter network traffic as you gather data or weed through a trace can save time and effort when analyzing a problem or proving the results.

My tool of choice is Wireshark. It uses two different types of filters, one for capture and the other for displaying data after you’ve captured the packets. Wireshark uses WinPcap, which provides link-layer network access, allowing all network packets to be filtered by the tool. This is done by bypassing the protocol stack which gives access to all the network traffic. It’s commonly used for many open source and commercial network tools. Gearbox’s deep packet capture engine appliance uses libpcap, a Linux version, to capture and filter packets due to its speed and efficiency.

One of the quickest ways to apply filters is to identify a packet and open up the detailed view of the packet (see Figure 1:1). After finding what you want to filter, right click and select Prepare a Filter>Selected. Then apply the newly desired filter now shown in the filter syntax box.

 

You can also use the display filters like capture filters (see Figure 1:2). First make sure the Update list of packets in real time option is selected. You’ll find it under Edit, select Preferences. Select the Capture option within the panel and look for Update list of packets in real time option.

 

Figure 1:2 Wireshark: How to select Display Filters

F2

 

For example, if you want to see what web sites people are viewing, select Domain Name System and you will see a DNS filter showing real time DNS packets.

When monitoring TCP transactions, look for the SYN and the SYN ACK responses. These responses measure the DELTA TIME from the SYN request to the response SYN ACK, giving you an accurate roundtrip time measurement.

Using tcp.options.mss_val returns all packets that replied to a SYN request, allowing you to view the MSS. You want this to be as large as possible. The maximum should be close to 1460 charters, which allows for maximum payload.

Another useful item to look for is the TCP Zero Window. When a server or load balancer is busy, it will send back a Zero Window. It shows when a device is too busy to receive a data packet. This can be done with tcp.analysis.zero_window filter.

Gearbit is a group of talented and experienced network experts providing tools and resources. As network analysts, we’re use to seeing capture filters in this format. So we gathered a list of Wireshark capture filters and organized them the way we like to view them. Visit the gearbit web site and download your own copy and add them to your Wireshark analyzer. You’ll be glad you did.

Under the Hood: Capture Filters, Display Filters – Part 2

In this month’s issue of Under-the-Hood, we’re continuing the discussion about the importance of filtering when using Wireshark. If you missed part 1 of the discussion, you can find the article “Under the Hood: Capture Filters, Display Filters” on the LoveMyTool website.

One way to improve and organize your Display Filters is to rearrange them into groups. At gearbit, we think like a network analyst, so we’ve compiled a list of capture filters and organized them by category. All you will need to do is copy the cfilter file containing the 150 filters that are organized by topic.

The compiled cfilters file has various types of filters that can be edited as you use them to meet most of your capture needs. If you find yourself adding additional filters, just follow the instructions listed that will show you how to create and organize your own filters.

The Capture Filters are found in the cfilters file located under C:\Documents and Settings\Administrator\Application Data\Wireshark [see Figure 1.1 Wireshark Capture Filter File Location]. Remember that the file folder “Administrator” may change according to how you have your logon setup. You can open the cfilters file in notepad for editing.

I like to rename the existing file so that I have a backup. The original file supplied with Wireshark doesn’t contain any type of format. We have found that by adding a few tabs to offset the filters from the Headers it gives the capture filters a whole new look, making the filters easier to find [see Figure 1.2 Wireshark cfilters File Format]. Adding categories also helps keep similar filters together.

The cfilters file at the gearbit web site provides a combination of Capture Filters that have been accumulated from several sources. By scouring various websites over several years we have compiled a master list which we then ran through a group of network experts for edits and additional input.. Additionally, network analysts like you continue to send us your favorite filters.. The list now contains more than 150 filters. If you have capture filters that you would like to contribute, please send them to tech08 at “@” gearbit.com.

Figure 1.1 Wireshark Capture Filter File Location

Figure 1_1 cfilters

The cfilter file can be edited, first organized into groups by Theme. To make the filters easier to see apply a Tab offsetting the test. All of the text needs to be contained in quotes. The Theme Titles needs to also have the word HEADER placed in the title.

Figure 1.2 Wireshark cfilters File Format

Figure 1_2 cfilters_file_format

Understanding the Need for Protocol Analysis

Network Protocol Map (available from Javvin)

 Protocol

 

I’m often asked, is bandwidth the most common cause of slow response? My answer to that is NO, most definitely not. In many cases throwing bandwidth at the problem is the common response and is very expensive. Over the years it proves to be the problem less than 5% of the time. So why not take a look at what is really happening on the wire?

From a piece of the transaction being slow that has a domino effect causing the whole transaction to suffer, or misconfiguration within servers or clients, the answer to these types of issues lies in looking through the packets. To understand what is taking place is a simple process in a complex system and with a huge variety of components. You can explain it all by investigating the transaction at the packet level.

You may ask what tools are needed? One of the main tools for this type of troubleshooting is a protocol analyzer. There are many available at a wide variety of prices. Fluke Optiview, Microsoft’s Netmon, Net Scout’s Network General Sniffer, Wildpackets, and WireShark are just a few of the many available.

WireShark which has been rebranded from the old name Ethereal has become a very powerful tool and has a universal flair. It has opened the door for many technical people to understand packet traces. It is now being utilized by program developers, engineering, support staff data and voice. No more the tool of just the network analyst.

If you have ever had a desire to understand more about how things really communicate within the network, or to explain to the server team or application group that it’s their problem with an accurate, clear, concise detail, well now you can. You will achieve pin point accuracy while precisely diagnosing the problem.

Then the answer is in getting the knowledge of network packets, protocols and how they are used in the network. Developing the knowledge at the packet level truly does just that. Understanding protocols and how they control speed, through put and their flow through the network controlling everything about the connection is the key.
Problem Remembered

#1 – A WAN where the router is forwarding packets that goes all the way to the destination (one side of the US to the other) only for the destination router to send it back. This happens until the time to live expires, but until this happens the packet is repeated 127 times. This “little problem” was consuming 40% of the WAN.

#2 – The problem in this case was slow response time because data packets were using the redundant WAN connection instead of the primary. We are talking about a circuit from Houston to Austin. The latency was so huge that it was extremely slow. The carrier trying to make sure that the path was redundant engineered the connections from Houston to Atlanta to Austin. I didn’t get it either but they explained; “To make sure that the fiber route between Houston and Austin was totally separate it went East out of Houston and back to Austin in a totally different direction. Not in the same ditch; as they explained. Now instead of going 185 miles, it traveled 1654 miles. Does the carrier not know how much delay they introduced? Round trip latency was 3.7 milliseconds now increased to a whopping 33.08 milliseconds.

#3 – The traffic was using the server disk backup network instead of the production network, allowing packets to be sent from the web server to the SQL database. The backup network had no IP address, instead it was setup with NetBEUI protocol. This happens when the web server has misconfigured a DNS address in the TCP/IP network interface and is not getting a response from a DNS request. The WEB server reverts to a broadcast and tries all interfaces. After getting a response from the NetBEUI interface it uses it. All of this takes extra time causing the HTTP sessions on the front end to perform slowly and timeout.

#4 – Or a routing loop on the VLAN where the corporate Exchange servers sit. Taking one packet and routing 128 times, over and over again. Oh, yea it does this for every packet sent to and from this VLAN. This does slow things down.
At Gearbit we offer training at the packet level that gives insight into the protocols that the network is built on. With this view you’re able to see how things really work, not just according to the manufacturer explanation. The end result you can now know for sure.

Under the Hood – Colorize TCP Session Conversations

Welcome to Under the Hood, gearbit’s newsletter for customers and network analysts. This technical tip we’ll reviewing the use of the Colorize Conversation rule, that allows you see TCP conversation by highlighting. Really, anything to help us clarify the trace as we’re analyzing is a huge plus.

You can access the Colorize Conversation Rule in a couple of ways.First way is through the Main Menu where you’ll find it listed under the View menu.
View>Colorize Conversation

Second is to select the packet, then right click, to bring up the Color Conversation rule.
Select Packet-Right Click>Colorize Conversation>Ethernet or IP or TCP

This is not to be confused with the Coloring Rule, that’s used to highlight a bit pattern or protocol, etc…. Look forward to a future article where we’ll take a look at several examples of how and when it can be used to speed up packet analysis.

 

 

In Figure 1:1 I’ve first used a Display filter to find the first packet of a TCP Session, buy using this display filter, (!(tcp.flags.ack == 1)) and (tcp.flags.syn == 1). This makes it easy to see all the TCP conversations.

A quick note, I just about always use the right click Prepare a Filter to build all of my Display Filters.

Now on to the Colorize Conversation rule. In the following examples, Figure 1:3 shows the packets highlighted in pink, showing a visual separated from all the other packets. With the packets now visible you can see the TCP conversation, but also see other network traffic that is occurring around and during this TCP session. Very help feature in Wireshark, thank you Gerald Comes and the Wireshark team!

In Figure 1:2 show how this is done, first selecting the View menu, and then the Colorize Conversation rule, along with the following menus that narrow down the conversation to the TCP Session.

Figure 1:1 TCP Sessions Displayed with TCP Flags Filter

TCP Flags SYN & Not ACK

 

Figure 1:2 Apply Colorize Conversation to TCP Session

Colorize Conversation for TCP Session

 

Figure 1:3 TCP Session Colorized

Under the Hood: Nagle Struck Again

 

Under-the-hood-bbq

As I imagine most of you are, I’m always looking for performance tuning ideas, something that really helps improves performance, preferably with minimal cost. Recently, while reviewing trace files, I discovered the cause of an issue I was troubleshooting: Nagle’s algorithm had struck again.

Don’t get me wrong, I’m not against Nagle or his algorithm, but I don’t like it when things work against you, like my wife’s car, which has an alarm system that locks all the doors, usually when you least expect it. I like the thought of safety, but lock all the doors but mine. I’ll lock my own door, thank you.

Nagle’s Congestion Control in IP/TCP Internetworks (RFC 896) describes what he called the “small packet problem,” where an application repeatedly emits data in small chunks, frequently only one byte in size. Since TCP packets have a 40 byte header (20 bytes for TCP, 20 bytes for IP) and Ethernet 18 byte (14 bytes source and destination, type and 4 bytes CRC), this results in a 59 byte packet for one byte of useful information, a huge and usually unnecessary overhead.

 

This situation often occurs in Telnet sessions, where most key presses generate a single byte of data that is transmitted immediately. Worse yet, over slow links, many such packets can be in transit at the same time, potentially leading to congestion collapse.

Just as my wife’s car locks all the doors, with good intentions, Nagle’s Congestion Control can have a negative effect. In Figure 1, a 200 millisecond delay interrupts the flow of data being transmitted. Using IO Graphs within Wireshark it become more obvious.

 

Figure 1: Wireshark trace indicating 200 millisecond delays, IP Graphs with 200 millisecond delays between data throughput spikesF1

Correcting the problem was straight forward by turning off Nagle’s setting, a registry setting with Microsoft.

One last word of wisdom: always monitor the results of any tuning configuration changes carefully to be absolutely sure that it has a positive impact. I always look through the trace file that is measuring the results to confirm that the change had a positive effect.