How to Troubleshoot IPSec VPN connectivity issues

Printer Friendly Page

This document is intended to help troubleshoot IPSec VPN connectivity issues. It is divided into two parts, one for each Phase of an IPSec VPN.

 

Phase 1

  • To rule out ISP-related issues, try pinging the peer IP from the PA external interface. Ensure that pings are enabled on the peer's external interface.
  • If pings have been blocked per security requirements, see if the other peer is responding to the main/aggressive mode messages, or the DPDs. Check for the responses of the "Are you there?" messages from the peer in the system logs under the Monitor tab or under ikemgr logs.
  • Check that the IKE identity is configured correctly.
  • Check that the policy is in place to permit IKE and IPSec applications. Usually this policy is not required if there is no clean-up rule configured on the box. If a clean-up rule is configured, the policy is configured usually from the external zone to the external zone.
  • Check that proposals are correct. If incorrect, logs about the mismatch can be found under the system logs, or by using the following CLI command:
    less mp-log ikemgr.log
  • Check that preshared key is correct. If incorrect, logs about the mismatch can be found under the system logs, or by using the following CLI command:
    less mp-log ikemgr.log
  • Take packet captures to analyze the traffic. Use filters to narrow the scope of the captured traffic.

  • Useful CLI commands:
> show vpn ike-sa gateway <name>
> test vpn ike-sa gateway <name>
> debug ike stat

Advanced CLI commands:

  • For detailed logging, turn on the logging level to debug:
> debug ike global on debug
> less mp-log ikemgr.log
  • To view the main/aggressive and quick mode negotiations, it is possible to turn on pcaps for capturing these negotiations. Messages 5 and 6 onwards in the main mode and all the packets in the quick mode have their data payload encrypted:
> debug ike pcap on
> view-pcap no-dns-lookup yes no-port-lookup yes debug-pcap ikemgr.pcap
  • Turn off debugs
> debug ike pcap off 

Configuring packet filter and captures restricts pcaps only to the one worked on, debug IKE pcap on shows pcaps for all VPN traffic.

 

To check if NAT-T is enabled, packets will be on port 4500 instead of 500 from the 5th and 6th messages of main mode.

Check if vendor id of the peer is supported on the Palo Alto Networks device and vice-versa.

 

Phase 2

  • Check if the firewalls are negotiating the tunnels, and ensure that 2 unidirectional SPIs exist:
> show vpn ipsec-sa
> show vpn ipsec-sa tunnel <tunnel.name>
  • Check if proposals are correct. If incorrect, logs about the mismatch can be found under the system logs under the monitor tab, or by using the following command:
less mp-log ikemgr.log
  • Check if pfs is enabled on both ends. If incorrect, logs about the mismatch can be found under the system logs under the monitor tab, or by using the command:
less mp-log ikemgr.log
  • Check the proxy-id configuration. This is usually not required when the tunnel is between two Palo Alto Networks firewalls, but when the peer is from another vendor, IDs usually need to be configured.
    A mismatch would be indicated under the system logs, or by using the command:
> less mp-log ikemgr.log
  • Useful CLI commands:
> show vpn flow name <tunnel.id/tunnel.name>
> show vpn flow name <tunnel.id/tunnel.name> | match bytes
  • Check if encapsulation and decapsulation bytes are increasing. If the firewall is passing traffic, then both values should be increasing.
> show vpn flow name <tunnel.id/tunnel.name> | match bytes

If encapsulation bytes are increasing and decapsulation is constant, then the firewall is sending  but not receiving packets.

  • Check to see if a policy is dropping the traffic, or if a port translating device in front of PAN that might be dropping the ESP packets.
> show vpn flow name <tunnel.id/tunnel.name> | match bytes

 

If decapsulation bytes are increasing and encapsulation is constant, then the firewall is receiving but not transmitting packets.

  • Check to see if a policy is dropping the traffic:
> test routing fib-lookup virtual-router default ip <destination IP>
--------------------------------------------------
runtime route lookup
--------------------------------------------------
virtual-router:  default
destination:      10.5.1.1
result:          interface tunnel.1

show routing route
> test vpn ipsec-sa tunnel <name>

Advanced CLI commands:
> debug ike global on debug
> less mp-log ikemgr.log
> debug ike pcap on
> view-pcap no-dns-lookup yes no-port-lookup yes debug-pcap ikemgr.pcap
> debug ike pcap off

If tunnels are up but traffic is not passing through the tunnel:

  • Check security policy and routing.
  • Check for any devices upstream that perform port-and-address-translations. Because ESP is a layer 3 protocol, ESP packets do not have port numbers. When such devices receive ESP packets, there is a high possibility they may silently drop them, because they do not see the port numbers to translate.
  • Apply debug packet filters, captures or logs, if necessary, to isolate the issue where the traffic is getting dropped.

 

owner: kprakash

Tags (10)
Comments

admin@firewall> less mp-log ikemgr

/var/log/pan/ikemgr: No such file or directory

Any ideas? Trying to get a tunnel up between my PA-200 running 6.1 and a VYOS box, it seems like there is absolutely no traffic passing, even though if I remove the VYOS host from my allow vpn rule, i'm seeing lots of deny messages but once I allow, no more messages.

I am sorry, the documentation was off just slightly..

The command needs to read:

> less mp-log ikemgr.log

If ever in doubt, use "tab" as it will autocomplete via the CLI.

Please try that and see if that provides you with any more information.

Thanks, unfortunately it didn't help me resolve my problem. I'm trying to setup a VPN tunnel between my PA-200 and a Vyos(a fork of Vyatta). Does the PAN OS restrict who it can bring up a tunnel with? I'm not having an issue with any other devices, Cisco, Watchguard, etc.

If you got a lot of deny messages until you have allowed the traffic.. then we could be looking at a routing issue also.  Please check on those routes and that they are pointing to the correct Tunnel interface on the Firewall.

I recommend opening a case with our Technical Support, as this could be very involved.

This is one of the best document to troubleshoot Ipsec VPN issues!

What if the tunnel status is showing up, still my traffic is not flowing through (though i have all the seccurity policies, NAT and routes in place) ? Sometimes i need to manually clear the IKE session id's on both the firewalls, then it starts working ..... is there any bug releted to this ? ( On one firewall i am using PAN OS 5.0.6, while the other one is running on PAN OS 6.0.10)

@R_Kushwaha, please contact our Technical Support for an issue like the one you've described. That will require the involvement of our support organization to troubleshoot

Excellent writeup.  Thank you.

One more thing for folks to check - from experience here:

 

* If decapsulation bytes are increasing and encapsulation is constant, then the firewall is receiving but not transmitting packets.

 

Double check that you don't have a PBF overrriding your static routes - the tunnel will get brought up, but the PBF will route the traffic.

Hi,

 

When I did less mp-log ikemgr.log, I get this error:

 

[PROTO_ERR] : received notify payload is not encrypted.

 

Any suggestions.

Thanks in advance.

 

Regards,

Raghav

Hi @RbadigerCY

as the log states the ikemgr received an unencrypted payload (from a remote source). you'll need to do a little troubleshooting  so you can match the eroor message to the source and then correct the configuration on the remote system

just had an issue with 2 ISP connection and the cusomte had configures static route for the peer thought the other ISP... this is how it looks like:

 

SECONDARY(active)> show session id 44444

Session 44444

c2s flow:
source: local [untrust]
dst: remote
proto: 17
sport: 500 dport: 500
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

s2c flow:
source: local [untrust]
dst: remote
proto: 17
sport: 500 dport: 500
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

start time : Mon Nov 20 09:09:33 2017
timeout : 600 sec
time to live : 580 sec
total byte count(c2s) : 81730
total byte count(s2c) : 45144
layer7 packet count(c2s) : 475
layer7 packet count(s2c) : 228
vsys : vsys1
application : ike
rule : IPSEC Establish
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
layer7 processing : enabled
URL filtering enabled : False
session via syn-cookies : False
session terminated on host : True
session traverses tunnel : False
captive portal session : False
ingress interface : ethernet1/2
egress interface : ethernet1/4
session QoS rule : N/A (class 4)
end-reason : unknown

@minow

 

Your post states that a firewall configured to use two different ISPs, presumeably where one is the primary and the other is the secondary. The customer configured a static route to reach the secondary ISP by going through the primary ISP.

 

 

Are you pointing out why it is not reasonable to use the primary ISP in order to reach the secondary ISP or are you trying to report that this configuration is causing a problem for the IPSec tunnel to be established?

 

The session detail shows that many packets have been exchanged. If there is a problem, you should contact Support to get assistance on this issue.

less mp-log ikemgr.log is a cumbersome command because if you have a long log file, it will take you a while to get to the end.

Instead, I use:

tail lines 100 mp-log ikemgr.log


 

@Mike_Eichin wrote 
less mp-log ikemgr.log is a cumbersome command because if you have a long log file, it will take you a while to get to the end

You can do shift+G to get to the end.

@Mike_Eichin , @mcunningham-cartera

 

Here's a good cheatsheet for Less:

 

Navigation  
SPACE forward one window
b backward one window
d forward half window
u backward half window
j navigate forward by one line
10j 10 lines forward.
k navigate backward by one line
10k 10 lines backward.
G go to the end of file
g go to the start of file
q or ZZ exit the less pager
Search  
/ search for a pattern which will take you to the next occurrence.
? search for a pattern which will take you to the previous occurrence.
n for next match in forward
N for previous match in backward
Other  
F simulate tail -f inside less pager
ma mark the current position with the letter ‘a’,
‘a go to the marked position ‘a’.
CTRL+G show the current file name along with line, byte and percentage statistics.