PA Ipsec 2/3 phase 2 coming up only

[JRussell]

Hi there,

I have a unique linux firewall box (that connects back to PA via Ipsec tunnels) on one of my sites. It is unique in the fact it requires 5 NIC's for the networks there. It only uses 3 phase 2 Ipsec tunnels which is the same on all my sites, but I have noticed some issues.

Namely that some of the time only 2 out of 3 tunnels come up. Sometimes all 3 come up, sometimes only 2. Fortunately the Main LAN always comes up, so users are not affected. Phase 1 tunnel comes up fine every time.

So I am trying to build a replacement box to test (as well as a backup in case the live one goes down), but when I boot the box up I get the Phase 1 come up fine. Then the main LAN Ipsec tunnel comes up. But for some reason it takes a very long time for that 2nd one to come up.

The original box was built by my predecessor and he left no documentation as to how he built this box.

I realize that this isn't necessarily a Palo problem, but it connects back to my PA firewall and all my other boxes of the same build are connecting without issue.

Any suggestions would be greatly appreciated. Is there perhaps some way I can monitor the Phase 2 portions of the Ipsec so that I can see what is happening? Apart from the system logs is there anywhere I can look to try help me identify the issue?

[kadak · Accepted solution]

Hello JRussell,

On the Palo Alto firewall, you monitor ikemgr.log while phase-2 is negotiating. For that you would need to setup the ike daemon to 'debug' level and tail follow the log.

> debug ike global on debug (make sure you give this command twice to make sure that the daemon logging is set at debug level rather than info level )

**************************************

> debug ike global on debug

sw.ikedaemon.debug.global: normal

> debug ike global on debug

sw.ikedaemon.debug.global: debug

**************************************

Once that is done, you can execute the following command:

> tail follow yes mp-log ikemgr.log

2014-06-19 14:34:16.194 -0500 ikemgr: panike_daemon phase 1 finished with status 1

2014-06-19 14:34:22.829 -0500 ikemgr: panike_daemon phase 2 started

2014-06-19 14:34:22.829 -0500 pan IKE cfg phase-2 triggered.

2014-06-19 14:34:22.829 -0500 pan IKE cfg phase-2 triggered when not necessary, skipped.

2014-06-19 14:34:22.829 -0500 ikemgr: panike_daemon phase 2 finished

2014-06-20 10:51:19 [INFO]: panike_debug_level_cb 4 => 5

2014-06-20 10:51:20.603 -0500 debug: ifmon_request_put(daemon/panike_sysd_if.c:916): 16 write to pipe: debug_level

2014-06-20 10:51:20.603 -0500 debug: ifmon_request_get(daemon/panike_sysd_if.c:932): 16 read from pipe, msg type 1

.

,

.

.

and so forth....

To turn off the debugging:

> debug ike global on normal

sw.ikedaemon.debug.global: debug

The following document helps troubleshooting VPN scenarios:

How to Troubleshoot VPN Connectivity Issues

Hope that gets your started!

Thanks and regards,

Kunal Adak

[JRussell]

Thanks Kunal,

I will give that a read. But just looking at that command. Won't that give me all phase 2 negotiations? I ask because that will give me an absolute tone of data returned as we have about 40 sites, all with 3 tunnels on them.

Is there any way to refine those Ike debug to only show the one site and it's tunnels?

[Rick_Rutherford]

Ideally, you would be able to pipe the results of the tail command to grep, but there's nothing in the PAN-OS command line reference doc that indicates that's possible.

Am I missing something that would allow you to filter the output of a tail command?

[pulukas]

You should be able to start with the system log events in the Monitor tab.  Filter for the ipsec messages here and see what top level error messages you might be getting from the failed connection.

[JRussell]

Somehow through doing all those troubleshooting guides and mucking around with the firewall settings on the other end I managed to get it up and running.

Thanks all for your suggestions!