Episode Transcript:
Angelo Eisma is a Senior Technical Support Engineer at Palo Alto Networks. He has previously worked for a Security Operations Center and a Telco. As part of Palo Alto Networks TAC, he is an SME for Remote Access and ID Management and is highly enthusiastic about sharing his knowledge and experience with customers.
John:
Hello, and welcome back to PANCast™. In today's episode, we will discuss Tools that TAC uses. We have a special guest today Angelo Eisma who will share more on this topic.
Before we get started, Angelo, could you tell us more about yourself?
Angelo:
Hi John, thank you for having me. I am quite excited because this is my first time joining your podcast. I've been with Palo Alto Networks for 5 years supporting our Next Generation Firewalls and today I introduce you to some of the tools we use when we troubleshoot an issue.
John:
Thank you Angelo, so what are some challenges when troubleshooting and how do these tools help?
Angelo:
While troubleshooting issues I would usually get more information than what the traffic logs or system logs can provide to isolate the issue.
If it is a transit traffic issue, I want to confirm that traffic was received and transmitted by the device processing the traffic.
If there is an issue about a certain firewall feature not working, like User-ID, I would check the logs for the User-ID process.
To do these we would use certain tools for a specific issue.
We first discuss tools for troubleshooting transit traffic issues which are Wireshark and tcpdump. These are powerful network analysis tools that enable capturing and analyzing network traffic in real time. They are used by network administrators, security professionals, and developers to troubleshoot network issues, analyze network performance, and detect security threats.
Wireshark, the network analysis tool
First, What is Wireshark? It is a graphical network analysis tool that provides a user-friendly interface for capturing and analyzing network traffic. It supports a wide range of network protocols, including TCP, UDP, HTTP, DNS, and more. It allows capturing and analyzing network traffic at a very granular level, down to the individual packet level. You can filter and search through network traffic to identify specific patterns or issues, and you can even visualize network traffic using graphs and charts.
When you have these capabilities, it is easier for you to visualize the traffic flow and inspect the contents of the packet to see if there are any issues with either the packet headers or the payload.
How does it work? As an example, when you have run a packet capture on Palo Alto Networks NGFW’s, you can download the capture files and load them into Wireshark. Once loaded, Wireshark will show you the network packets captured in the file and from here you can analyze the communication. From here, you can perform the following to analyze the packets as you see them without using the statistical tools built into Wireshark:
If the connection is TCP, check if the TCP 3-way handshake is present
You can check the packet length, and you can inspect the payload if the packet is not encrypted.
There are statistical tools available as well if you need to check the latency, conversations, or IP address statistics. These are not the only tools available in Wireshark and we recommend that you familiarize yourself with the other tools.
Apart from loading a capture file, you can also run Wireshark to capture packets. When you open the application it will prompt you to select an interface on which to capture packets and you can also use filters such as “host 8.8.8.8” to filter the packets being captured.
To install Wireshark, you can follow the installation guide from the Wireshark developer.
John:
That is a good overview of how Wireshark can help. Could you explain more about TCPdump?
TCPdump, a CLI network analysis tool
Angelo:
tcpdump is a command-line network analysis tool that provides a more lightweight and flexible way of capturing network traffic compared to Wireshark. Like Wireshark, It allows capturing network traffic in real time and saving it to a file for later analysis. It also supports a wide range of options and filters that allow capturing specific types of network traffic or filtering out unwanted traffic. But unlike Wireshark, tcpdump does not support graphs and charts. Tcpdump is also not directly available on Windows.
tcpdump, similar to Wireshark, can capture packets and save them to a file or it can also load saved packet capture files. The difference is that tcpdump has no GUI and all operations are done through the CLI.
How would you install TCPDUMP? It may or may not be installed on your computer. On Mac devices, it should be installed as well as on some Linux distributions (check the documentation on your Linux distribution for more information). On Windows, you would use Wireshark in place of tcpdump.
To install tcpdump on Linux machines, you will need to check if your Linux distribution repository contains tcpdump. Refer to your Linux distribution documentation or resources and also refer to the installation notes from the tcpdump website.
To use tcpdump, simply open a terminal window and enter the command followed by the desired filters and options. For example:
tcpdump -i eth0 tcp port 80
This will capture traffic on interface eth0 with source or destination port 80.
On Palo Alto Networks devices the syntax of tcpdump is different as shown by the example below and is used for the management interface only.
tcpdump filter "net 192.168.0.0/24 and port 80"
This will capture traffic with either the source or destination IP address within the 192.168.0.0/24 subnet with the source or destination port of 80.
To view the capture file use the following command.
view-pcap mgmt-pcap mgmt.pcap
Sometimes your packet captures might be truncated. When packets are truncated it might not contain enough information for analysis. In this case, use the option of “snaplen 0” to capture the whole packet.
John:
Thank you Angelo, you mentioned earlier a tool called less. Can you tell us more about this?
Less, the terminal pager
Angelo:
Less is a powerful terminal pager that allows you to view, navigate, and search through large files quickly and efficiently. It supports features like scrolling, searching, jumping to specific line numbers, and searching using regular expressions. The less command on Palo Alto Networks devices is similar to the less command on Unix systems.
To run the command, simply open a terminal window and enter the command followed by the name of the file you want to view. For example:
less mp-log ikemgr.log
less dp-log pan_packet_diag.log
less s2dp0-log pan_packet_diag.log
In Palo Alto Networks devices, the first command will open the file named “ikemgr.log” in the directory “mp-log”. Other directories exist specially for devices with a separate DP.
Less is already installed on all PAN devices running PAN-OS. No need for installation instructions.
Once you have opened the file in "less", you can use the arrow keys or the Page Up/Page Down keys to scroll up and down through the file. As you scroll, you will notice that "less" displays the file one screenful at a time, making it easy to read and navigate.
Another useful feature of "less" is its ability to search for specific text within the file. To search, simply press the "/" key, and then type in the text you want to search for. Press "Enter" to perform the search, and then use the "n" key to move to the next occurrence of the text, or the "N" (shift + n) key to move to the previous occurrence. You can combine this search function with regular expressions for an even more powerful search tool.
In addition to searching, "less" also allows you to jump to specific parts of the file using line numbers. To jump to a specific line, simply type the line number followed by the "G" key. For example, to jump to line 50, you would type "50G".
One of the most powerful features of "less" is its ability to handle large files efficiently. Unlike other text editors, "less" only loads the portions of the file that are being displayed on the screen, making it much faster and more memory-efficient. This makes it a great choice for viewing and navigating through very large files.
Then, when you have finished viewing the file, simply press the "q" key to quit "less" and return to the command line.
John:
This is good information to know on these tools. Can you explain more on Grep and regular expressions?
Grep and Regular Expressions, for pattern search in logs
Angelo:
Grep allows you to search for specific patterns of text within files, and it can be used to find words, phrases, or even complex patterns of text using regular expressions. Regular expressions or regex, are a powerful pattern-matching language that allows you to search for specific patterns of text, regardless of their exact wording. This makes grep an incredibly versatile tool for text processing and analysis.
In addition to searching for text, grep also supports various options and filters that allow you to refine your search and display only the relevant results. For example, you can use the "-i" flag to make the search case-insensitive, or use the "-n" flag to display line numbers for the matching results. Grep can also search for text within specific files or directories, or even within the output of other commands using pipes.
Regular expressions are supported by many Linux utilities, including grep, sed, and awk. For example, to search for all occurrences of the word "hello" regardless of case in the file “example.txt”, you could use the following regular expression with grep:
grep -i "\<hello\>" example.txt
In this regular expression, "<" and ">" are word boundaries, and the "-i" flag makes the search case-insensitive.
The example above shows the combined use of grep and regular expressions but the grep command on Palo Alto Networks devices do not support regular expressions. The equivalent command on Palo Alto Networks devices is shown below. Spaces can be used. In this regard, you are limited to searching for the exact string that you enter.
grep pattern "the quick brown fox jumps over the lazy dog" mp-log ms.log
Even though regular expressions are not supported in the grep CLI command of Palo Alto Networks devices, you can download and extract the tech support file on a system (e.g. Linux) which supports grep and use regular expressions or use an application which supports both (e.g. Cygwin).
John:
These independent tools seem very useful to help narrow down and identify a problem. Angelo - what would be the key takeaways for today?
Episode Key Takeaways
Angelo:
First we discussed Wireshark. A tool that can capture packets so that we can perform in depth analysis on the data it contains.
Then we have tcpdump. Similar to Wireshark it can capture packets but it is run through the CLI but does not have the statistical tools of Wireshark.
Next is less. You would use it in a CLI only environment to see the contents of a text file.
Then we discussed grep which can be used to search for specific string patterns within a text file.
And finally, regular expressions which can be used in conjunction with the search function in less and grep to create complex search string patterns.
By knowing how to use these tools, it will give you the capability to identify and fix issues.
John:
Thank you, Angelo, for sharing about these TAC tools for troubleshooting. You can find the transcript, the commands to run and some valuable links on live.paloaltonetworks.com under PANCast.
Angelo:
Thank you for having me, John. It was a pleasure to join you.
John:
PANCasters, if you have topics you need us to cover, please send in your feedback through the PANCast Ideas Submission page on LIVEcommunity, and we’ll be happy to review them.
Until next time. Bye!
Related Content:
NGFW Panorama
