GP disconnect intermittely

[MohamedSharief]

Hi All,

The users are able to connect and work for sometime and then traffic flow stops; although Global Protect Agent shows that VPN is connected.

The user have to disconnect and reconnect again, then it will work for sometime and stops again.

This intermittent issue can occur in 10 min, or 30 min or 1 hour or more, there is no definite time.

Initially, this happened with Global Protect VPN 3.1.4 and then we upgraded the agent to 3.1.6. This issue still persists.

When I took the debug logs from client I wasen't able to find any useful logs or maybe I didn't understand the logs correctly.

The logs from Firewall also is not showing anything.

Client is using PANOS 7.1.8 and GP agent 3.1.6

The portal and Gateway are on the same interface.

Below are logs from GP client:

(T4256) 04/05/17 19:14:11:862 Debug( 329): CheckHip over

(T4256) 04/05/17 19:14:11:862 Debug( 277): Hip checking is not initiated by clicking resubmit host profile.

(T4256) 04/05/17 19:14:11:862 Debug( 219): HipCheckThread: wait for hip check event for 3600000 ms);

(T7320) 04/05/17 19:14:14:347 Debug( 438): HipMissingPatchThread: now is 1491408854, last hip check is 1491408833, hip check interval is 3600000

(T7320) 04/05/17 19:14:14:347 Debug( 443): HipMissingPatchThread: wait 3565000 ms

(T7320) 04/05/17 20:13:38:790 Debug( 459): HipMissingPatchThread: WAIT_TIMEOUT

(T7320) 04/05/17 20:13:38:790 Debug( 377): CheckHipMissingPatchInOtherProcess()

(T7320) 04/05/17 20:13:38:790 Debug( 380): Need to check missing patch.

(T7320) 04/05/17 20:13:38:790 Debug(  76): pan_get_full_path(): full path in multibyte char is C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHipMp.exe

(T7320) 04/05/17 20:13:38:790 Debug( 301): CheckHipMissingPatchInOtherProcess(): Starting process PanGpHipMp.exe

(T7320) 04/05/17 20:13:49:373 Debug( 324): PanGpHipMp.exe exit for checking misssing patches.

(T7320) 04/05/17 20:13:49:373 Debug( 387): CheckHipMissingPatchInOtherProcess(): exits.

(T7320) 04/05/17 20:13:49:373 Debug( 474): Hip missing patch checking duration is 11

(T7320) 04/05/17 20:14:07:378 Debug( 438): HipMissingPatchThread: now is 1491412447, last hip check is 1491408833, hip check interval is 3600000

(T7320) 04/05/17 20:14:07:378 Debug( 443): HipMissingPatchThread: wait -32000 ms

(T7320) 04/05/17 20:14:07:378 Debug( 467): nSleep <= 0. m_tLastHipCheckEventWakeup is 1491408833, m_dwHipCheckInterval is 3600000, Now is 1491412447.

(T7320) 04/05/17 20:14:07:378 Debug( 377): CheckHipMissingPatchInOtherProcess()

(T7320) 04/05/17 20:14:07:378 Debug( 380): Need to check missing patch.

(T7320) 04/05/17 20:14:07:378 Debug(  76): pan_get_full_path(): full path in multibyte char is C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHipMp.exe

(T7320) 04/05/17 20:14:07:378 Debug( 301): CheckHipMissingPatchInOtherProcess(): Starting process PanGpHipMp.exe

(T4256) 04/05/17 20:14:11:317 Info ( 230): HipCheckThread: got check hip event or time out.

(T4256) 04/05/17 20:14:11:317 Debug( 242): HipCheckThread: WAIT_TIMEOUT

(T4256) 04/05/17 20:14:11:317 Debug( 762): SetNextScheduledHipCheckTime to 1491416051

(T4256) 04/05/17 20:14:11:317 Debug( 260): Last hip check event wakeup tick is 1491412451

(T4256) 04/05/17 20:14:11:317 Debug( 262): HipCheckThread: check hip in other process.

(T4256) 04/05/17 20:14:11:317 Debug( 301): CheckHipInOtherProcess()

(T4256) 04/05/17 20:14:11:317 Debug( 305): Need to collect hip data

(T4256) 04/05/17 20:14:11:317 Debug(  76): pan_get_full_path(): full path in multibyte char is C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHip.exe

(T4256) 04/05/17 20:14:11:317 Debug( 125): Starting process PanGpHip.exe

(T4256) 04/05/17 20:14:11:317 Debug( 142): Wait for the ready event of hip report generated in other process.

(T7320) 04/05/17 20:14:12:723 Debug( 324): PanGpHipMp.exe exit for checking misssing patches.

(T7320) 04/05/17 20:14:12:723 Debug( 387): CheckHipMissingPatchInOtherProcess(): exits.

(T7320) 04/05/17 20:14:12:723 Debug( 474): Hip missing patch checking duration is 5

(T7320) 04/05/17 20:14:26:726 Debug( 438): HipMissingPatchThread: now is 1491412466, last hip check is 1491412451, hip check interval is 3600000

(T7320) 04/05/17 20:14:26:726 Debug( 443): HipMissingPatchThread: wait 3571000 ms

(T4256) 04/05/17 20:14:33:993 Debug( 146): Got hip report in other process ready event.

(T4256) 04/05/17 20:14:33:993 Debug( 165): Read output from PanGpHip.exe

(T4256) 04/05/17 20:14:33:993 Debug( 202): write hip file now

(T4256) 04/05/17 20:14:33:993 Debug( 213): CheckHipInOtherProcess() sets hip report ready event.

(T4256) 04/05/17 20:14:33:993 Debug( 142): Wait for the ready event of hip report generated in other process.

(T14240) 04/05/17 20:14:33:993 Debug(3693): HipReportThread: got HIP report ready event.

(T14240) 04/05/17 20:14:33:993 Debug(3709): HipReportThread: wait for network discover ready event.

(T14240) 04/05/17 20:14:33:993 Debug(3714): HipReportThread: got network discover ready event.

(T14240) 04/05/17 20:14:34:008 Debug( 794): GetNicInfo(): NIC count is 9.

(T14240) 04/05/17 20:14:34:008 Debug(3745): Sending hip report delay max registry setting is -1 seconds

(T14240) 04/05/17 20:14:34:008 Debug(3747): Set max sending hip report delay to default 1800 seconds

(T14240) 04/05/17 20:14:34:008 Debug(3762): hip report is encoded

(T14240) 04/05/17 20:14:34:008 Debug(3784): HIP report md5 digest is c03a6ad93f88b6f247221b1cc1aa8e2

(T14240) 04/05/17 20:14:34:008 Debug(3810): HipReportThread: network type is external network.

(T14240) 04/05/17 20:14:34:008 Debug(3304): Entering SendHipReportToGateway(). Gateway: 195.226.x.2

(T14240) 04/05/17 20:14:34:008 Debug( 774): m_bScheduleFlag is 1

(T14240) 04/05/17 20:14:34:008 Debug( 784): m_bScheduleFlag is set to 1

(T14240) 04/05/17 20:14:34:008 Debug(3328): Gateway 195.226.x.2: now is 1491412474, next hip checking is 1491416051, next hip report check sending time is 1491412450, last hip report check sending time is 1491408850, sending hip delay is 0 ms

(T14240) 04/05/17 20:14:34:008 Debug(3345): Wait for 0 ms to send hip report check to gateway 195.226.x.2

(T14240) 04/05/17 20:14:34:008 Debug(3358): Time to send hip report to gateway 195.226.x.2

(T14240) 04/05/17 20:14:34:008 Debug(3370): Hip report head to gateway 195.226.x.2 is

<?xml version="1.0" encoding="UTF-8"?>

<hip-report>

                <md5-sum>c03a6ad93f88b6f247221b1cc1aa8e2</md5-sum>

                <user-name>I.Nawaz</user-name>

                <domain></domain>

                <host-name>H-B1-MIS-000-01</host-name>

                <host-id>de7d7c11-653e-4ae8-af54-aaa546543a96</host-id>

                <ip-address>10.212.134.203</ip-address>

                <gener

(T14240) 04/05/17 20:14:34:008 Debug(3506): SendHipReportNReceive()

(T14240) 04/05/17 20:14:34:008 Debug(3528): bUseCCUser=0, ccUserName=, m_userName=I.Nawaz

(T14240) 04/05/17 20:14:34:008 Debug(3530): using https to send hip report check to gateway 195.226.x.2

(T14240) 04/05/17 20:14:34:008 Debug(3565): Network discover SN 234 remains same.

(T14240) 04/05/17 20:14:34:008 Debug(3156): entering...

(T14240) 04/05/17 20:14:34:008 Debug(  76): pan_get_full_path(): full path in multibyte char is C:\Program Files\Palo Alto Networks\GlobalProtect\cc.pfx

(T14240) 04/05/17 20:14:34:008 Info (1259): File C:\Program Files\Palo Alto Networks\GlobalProtect\cc.pfx does not exist.

(T14240) 04/05/17 20:14:34:008 Debug(  47): WSAGetLastError() returns 10035

(T4256) 04/05/17 20:14:34:086 Debug( 146): Got hip report in other process ready event.

(T4256) 04/05/17 20:14:34:086 Debug( 165): Read output from PanGpHip.exe

(T4256) 04/05/17 20:14:34:086 Debug( 202): write hip file now

(T4256) 04/05/17 20:14:34:086 Debug( 213): CheckHipInOtherProcess() sets hip report ready event.

(T4256) 04/05/17 20:14:34:086 Debug( 142): Wait for the ready event of hip report generated in other process.

(T14240) 04/05/17 20:14:34:196 Debug(3169): CPanMSService::SendNReceive(): SSL is connected.

(T14240) 04/05/17 20:14:34:196 Debug(3187): Msg length is 378. Sending POST /ssl-vpn/hipreportcheck.esp HTTP/1.1

Accept: */*

Content-Length: 205

Connection: Keep-Alive

Content-Type: application/x-www-form-urlencoded

Host: 195.226.x.2

user=I.Nawaz&domain=alsalam&portal=SIH-GW-N&authcookie=********************************&client-ip=10.212.134.203&computer=H-B1-MIS-000-01&md5=c03a6ad93f88b6f247221b1cc1aa8e2&client-role=global-protect-full

(T14240) 04/05/17 20:14:34:274 Debug(1189): SSL3 alert write:warning:close notify

(T14240) 04/05/17 20:14:34:274 Debug(3577): Gateway 195.226.x.2, response to the hip report check:

                <response status="success">

                                <hip-report-needed>no</hip-report-needed>

                </response>

(T14240) 04/05/17 20:14:34:274 Info (3579): sent HIP report check to 195.226.x.2.

(T14240) 04/05/17 20:14:34:274 Debug(3607): Response status of HIP report check is success, gateway 195.226.x.2

(T14240) 04/05/17 20:14:34:274 Debug(3609): Hip report check returns success.

(T14240) 04/05/17 20:14:34:274 Debug(3384): SendHipReportNReceive returns TRUE for gateway 195.226.x.2

(T14240) 04/05/17 20:14:34:274 Debug(3397): Hip notification is empty in the HIP report check response from gateway 195.226.x.2

(T14240) 04/05/17 20:14:34:274 Info (3406): Hip report is not  needed for gateway 195.226.x.2.

(T14240) 04/05/17 20:14:34:274 Debug(3437): SSL is disconnected. Returns TRUE.

(T14240) 04/05/17 20:14:34:274 Debug(1048): SendHipReportToGateway 195.226.x.2 returns TRUE.

(T14240) 04/05/17 20:14:34:289 Debug( 794): GetNicInfo(): NIC count is 9.

(T14240) 04/05/17 20:14:34:289 Debug( 641): Hip report changed. Include it in status message to client.

(T14240) 04/05/17 20:14:34:289 Debug(3688): HipReportThread: wait for HIP report ready event.

(T14240) 04/05/17 20:14:34:289 Debug(3693): HipReportThread: got HIP report ready event.

(T14240) 04/05/17 20:14:34:289 Debug(3709): HipReportThread: wait for network discover ready event.

(T14240) 04/05/17 20:14:34:289 Debug(3714): HipReportThread: got network discover ready event.

(T14240) 04/05/17 20:14:34:305 Debug( 794): GetNicInfo(): NIC count is 9.

(T14240) 04/05/17 20:14:34:305 Debug(3745): Sending hip report delay max registry setting is -1 seconds

(T14240) 04/05/17 20:14:34:305 Debug(3747): Set max sending hip report delay to default 1800 seconds

(T14240) 04/05/17 20:14:34:305 Debug(3762): hip report is encoded

(T14240) 04/05/17 20:14:34:305 Debug(3784): HIP report md5 digest is c03a6ad93f88b6f247221b1cc1aa8e2

(T14240) 04/05/17 20:14:34:305 Debug(3810): HipReportThread: network type is external network.

(T14240) 04/05/17 20:14:34:305 Debug(3304): Entering SendHipReportToGateway(). Gateway: 195.226.x.2

(T14240) 04/05/17 20:14:34:305 Debug( 774): m_bScheduleFlag is 1

(T14240) 04/05/17 20:14:34:305 Debug( 784): m_bScheduleFlag is set to 1

(T14240) 04/05/17 20:14:34:305 Debug(3328): Gateway 195.226.x.2: now is 1491412474, next hip checking is 1491416051, next hip report check sending time is 1491416074, last hip report check sending time is 1491412474, sending hip delay is 0 ms

(T14240) 04/05/17 20:14:34:305 Debug(3342): dwWaitTime 3600000 exceeds max value 1800000. Set dwWaitTime to the max value

(T14240) 04/05/17 20:14:34:305 Debug(3345): Wait for 1800000 ms to send hip report check to gateway 195.226.x.2

(T4256) 04/05/17 20:14:35:086 Debug( 150): Got event for PanGpHip process has quited.

(T4256) 04/05/17 20:14:35:086 Debug( 329): CheckHip over

(T4256) 04/05/17 20:14:35:086 Debug( 277): Hip checking is not initiated by clicking resubmit host profile.

(T4256) 04/05/17 20:14:35:086 Debug( 219): HipCheckThread: wait for hip check event for 3600000 ms);

Hope anyone have a solution for this.

Regards,

Sharief

[MohamedSharief]

Anyone experienced the same behavior before?

[ansharma · Accepted solution]

Hi MohamedSharief,

Do you have security policies based on Users/groups? If so, one possible explanation would be that the user-ip mapping learned via GP is getting overwritten by UIA or something else. This eventually times out or may not be in the correct format. It wouldn't match what's in the security rule and hence the traffic would stop. Check the traffic logs and search by IP. Alternatively, once the users's traffic is not flowing, check the user-ip mapping 'show user ip-user-mapping ip x.x.x.x'.

If this does happen to be the case, use the Include/Exclude networks to exclude the GP users' subnet.

Regards,

Anurag

[MohamedSharief]

Hi @ansharma,

Pretty good explanation 🙂

I'll try and let you know the findings.

Regards,

Sharief

[arunkumar.duraisamy]

Hi Ansharma

We are also facing the same issue now. But we do not have any policies with usernames/group mapping. And this is happening for only one user, we have almost 300+ users are connected to GP-VPN.

Please confirm if we have any other issue found for this behavior.

[Retired Member]

Hi,

We have the same issue here with PAN OS 9.1.9 and GP 5.1.5.

As anyone of you guys solved the problem on your side?

[Aleksej.Filster]

Hi,

i have PanOS 9.1.4 and GP 5.2.5. And this issue. I would be very happy to have a solution

[Retired Member]

Hi,

I was finally able to reproduce the problem. Actually, we have an inventory tool and several times a day it scans the clients like after a software update. When this happen with a VPN client, it loose the connection but the GP client is still connected.

Do you think you could have something similar in your side?