Troubleshooting tools

[infotech]

I am new to firewalls and new to PA so I really need to find some tools and technique to be able to troubleshoot issues on my PA

[reaper]

Hi

A good start would be here : Education and Training - Palo Alto Networks Courses

There are several interesting courses that could come in handy including a free online e-learning "configuration 101"

hope this helps

Tom

[infotech]

I did the 101 already and it was helpfull and I looked on the online courses but I was hoping to find some free stuff and reference material to always have on hand

[_slv_]

So second step is to read this community and learn from us. Beleave me - it's working.

I'm a good example, I didn't pass any PA courses or exams, but after two years I have a good community score and knowledge

Regards

SLawek

[DaveCorwin]

Here is a good list of CLI commands to help you out:

General system health

·         show system info –provides the system’s management IP, serial number and code version

·         show system statistics – shows the real time throughput on the device

·         show system software status – shows whether various system processes are running

·         show jobs processed – used to see when commits, downloads, upgrades, etc. are completed

·         show jobs all -show any jobs in progress

·         show job id <id#> -to show any warning/error in configuration

·         clear job id <id#> -to clear a hung job

·         show system disk-space- show percent usage of disk partitions

·         show system logdb-quota – shows the maximum log file sizes

·         debug dataplane internal vif link – show management interface (eth0) counters

·         show system state filter cfg.general.max* - To display the System Limits for objects, profiles, and policies

To monitor CPUs

·         show system resources -  shows processes running in the management plane similar to “top” command

·         show running resource-monitor – used to see the resource utilization in the data plane, such as dataplane CPU utilization

·         less mp-log mp-monitor.log – Every 15 minutes the system runs a script to monitor management plane resource usage, output is stored in this file.

·         less dp-log dp-monitor.log - Every 15 minutes the system runs a script to monitor dataplane resource usage, output is stored in this file.

General dropped packet troubleshooting

·         ping source <IP_addr_src_int> host <IP_addr_host> - allows to ping from the specified FW source interface

·         ping host <IP> - ping from the MGT interface

·         show session all | match – used to show specific sessions in the session table.  You can enter any text after the word match.  A good example would be a source or destination IP or an application

·         show session all | filter destination <IP> dest- shows all sessions going to a particular dest IP and port <port>- port

·         show session all filter type predict – To show any pin-hole applications (e.g.FTP)

·         show session id – shows the specifics behind a particular session by entering the ID number after the word “id”

·         show counter interface – shows interface counters

·         show counter global | match drop – used to troubleshoot dropped packets

·         show counter global delta yes | match [source ip|dest ip| drop | error  | frag ] – show counter changes since last time ran this command, filter on particular keyword

·         show counter global filter packet-filter yes delta yes – show counter changes since last time ran this command, filter on debug filter

·         show counter global filter delta yes – show counter changes since last time ran this command

NAT

·         show running nat-policy- shows current NAT policy table

·         show running ippool- use to see if NAT pool leak

·         test nat-policy-match – simulate traffic going through the device, what NAT policy will it match?

Routing

·         show routing route – displays the routing table

·         test routing fib-lookup virtual-router <VR_name> ip <IP_addr_trying_reach> - finds which route in the routing table will be used to reach the IP address that you are testing

Routing Debug Commands

·         debug routing global on debug

·         less mp-log routed.log - To view the log

·         tail follow yes mp-log routed.log - To view the log in real time

Policies

·         show running security-policy – shows the current policy set

·         test security-policy-match from trust to untrust destination <IP>- simulate a packet going through the system, which policy will it match?

PAN Agent

·         show user pan-agent statistics – used to see if the agent is connected and operational.  Status should be connected OK and you should see numbers under users, groups and IPs.

·         show pan-agent user-IDs  - used to see if the FW has pulled groups from the PANAgent

·         show user ip-user-mapping – used to see IP to username mappings on the FW

·         clear user-cache all – clears the user-ID cache

·         debug device-server reset pan-agent <name> - reset the firewall’s connection to the specified agent

URL

·         request url-filtering upgrade brightcloud- If URL does not show up on dynamic updates page run this command

·         test url <url or IP> – used to test the categorization of a URL on the FW

·         tail follow yes mp-log pan_bc_download.log – shows the BrightCloud database update logs

·         request url-filtering download status – shows the status of the database download (essentially the very last line from the pan_bc_download.log file)

·         debug dataplane show url-cache statistics– shows statistics on the URL cache

·         show counter global | match url – shows statistics on URL processing

·         clear url-cache – used to clear the URL cache- cache contains 100k of the most popular URLs on this network

·         show log url direction equal backward- view the URL log, most recent entries first

·         To test connectivity to the BrightCloud servers:

o   ping host service.brightcloud.com

o   ping host database.brightcloud.com

Log viewing / deleting[1]

·         show log [ system | traffic | threat ] direction equal backward – will take you to the end of the specified log

·         show log [ system | traffic | threat ] direction equal forward – will take you to beginning of the specified log

·         clear log [ traffic | threat | acc ] – clear everything in the specified log

·         show log traffic receive_time in ? - pick a timeframe from the list

·         sho log traffic app equal gmail - show only gmail traffic in log

IPSec

·         To view detailed debug information for IPSec tunneling:

1.      debug ike global on debug

2.      less mp-log ikemgr.log

3.      test vpn ike-sa gateway <gw_name> - initiates traffic to bring up tunnel

4.      show vpn ike-sa gateway <gw_name> - to see if phase 1 is up

5.      show vpn ipsec-sa tunnel <tunnel name> - to see if phase 2 is up

6.      show vpn flow – to see all active tunnels

7.      sho vpn flow <name> or tunnel-id <id#>  -to see detailed info on the tunnel

HA

·         show high-availability state – shows the HA state of the FW you are on

·         show high-availability state-synchronization – shows if the FWs are synced

·         show high-availability path-monitoring – shows the status of path monitoring

·         request high-availability state suspend – this will suspend active box and make the current passive device active

·         request high-availability clear-alarm-led – this will clear the HA failover alarm on the unit

Vsys

·         set system setting target-vsys <vsys #> -to enter a vsys

·         set system setting target-vsys none – to exit a vsys

Software, Content, and Licenses

·         To upgrade the software on the FW:

1.      tftp import software from <IP_addr_tftp_server> file <filename>

2.      request system software install file <filename>

3.      request restart system

·         request system software [info | check | download | install ] –manipulate PANOS software from the CLI

·         To upgrade the content on the FW:

1.      tftp import content from <IP_addr_tftp_server> file <filename>

2.      request content upgrade install file <filename>

·         request content downgrade install previous –downgrade to the previous content version

·         request system private-data-reset- to clear config and logs/reports

·         debug swm [ status | list | revert ] – will show possible code to install, or code that was installed. “revert” is used to revert to last running OS version without having to do a factory reset (such as from 4.0 to back to 3.1)

·         request license info – shows the license installed on the device

·         delete license key ? – use to delete a license file if having issues and want to retrieve new licenses, use question mark to list file names, only delete the files you see fit

Config diff/force/cli format

·         show config diff- compares two versions of the config

·         commit force- perform a commit, even if there are errors

·         set cli config-output-format set- use to view the config in “set” format from within the configure prompt (#)

Misc

·         set deviceconfig setting session tcp-reject-non-syn no – used to ignore SYN when creating sessions; confirm command took effect with show session info

·         set deviceconfig setting session offload no –- makes all packets go through CPU, otherwise all fastpath packets just go through EZ chip (turns off session offload to fastpath); confirm command took effect with show session info

·         set deviceconfig setting tcp drop-out-of-wnd <yes|no>; confirm command took effect with show running tcp state

·         debug dataplane pool statistics - this will show the different dataplane buffers and can be used to see if the system is nearing capacity in certain functionality.

·         show system state filter sys.s(x).p(x).phy -command to see physical media

·         set cli pager off - To disable the more function

·         delete network interface ethernet ethernet1/x- deletes any setting on the interface

·         request system private-data-reset- delete private data but keeps software,content installations

·         show system files- to see if FW generated any core-files

·         grep mp-log * pattern (what your searching for-name)- to search all logs for a specific word

·         less dp0-log brdagent.log- to check to see if you have physical errors on interface

·         less dp0-log mprelay.log- to check to see if you have physical errors on interface

·         show system state filter-pretty sw.comm.s1.*.session-info | match active- to see number of sessions on each data-plane

·         https://x.x.x.x/esp/restapi.esp?type=keygen&user=admin&password=admin – To generate a API key

Debug Commands

·         debug dataplane packet-diag show setting - to see if any filters or capture are set

·         debug dataplane packet-diag set filter on - to turn on filter

·         debug dataplane packet-diag set filter match source x.x.x.x destination x.x.x.x destination-port X file test.pcap

·         debug dataplane packet-diag set capture stage <receive,drop,firewall,transmit> file <file name>

·         debug dataplane packet-diag set caprture on - to turn capture on

·         view-pcap follow yes <filter-pcap,debug-pcap> test.pcap yes- this allows you to view the data real time

·         view-pcap filter-pcap <file name>

CLEAN UP COMMANDS:

·         debug dataplane packet-diag set capture off - to stop capturing data

·         debug dataplane packet-diag set filter off- shut off filter

·         delete debug-filter test.pcap - to delete the file

Debug Flow Basic

·         debug dataplane packet-diag filter on

·         debug dataplane packet-diag set filter source x.x.x.x dest y.y.y.y

·         debug dataplane packet-diag set log on

·         Generate traffic

·         less dp0-log pan_packet_diag.log

CLEAN UP COMMANDS:

·         debug dataplane packet-diag clear log log

·         debug dataplane packet-diag filter off

·         debug dataplane packet-diag set log off

---

[1] Arguments that are shown with square braces and pipe symbol mean that you choose one of the arguments listed. For example, [ arg1 | arg2 | arg3 ] means you select either “arg1” or “arg2” or “arg3”.

[infotech]

Awesome thanks and knowing how to interpret what I find would be really useful.

[pulukas]

The troubleshooting collection from Support:

Troubleshooting Palo Alto Networks Hardware Issues

Troubleshooting User-ID: Group and User-to-IP Mapping

Packet Based Troubleshooting - Configuring Packet Captures and Debug Logs

Troubleshooting User Activity Reports

Troubleshooting GlobalProtect, PAN-OS 4.1

How to Troubleshoot VPN Connectivity Issues

Understanding PAN-OS NAT

[infotech]

Awesome thanks