Global Protect Portal Failures

[inclusa-admin]

Our organization has started noticing that every 24 hours (give or take an hour) new connections to our Global Protect VPN service is rejecting new connections to the appliance. We also notice that the portal landing page stops responding and issues a generic SSL Error and the page cannot be displayed. The issue is only resolved by rebooting the firewall.

We are running an Azure VM-300 and have roughly 1000 users connecting throughout the day. 

No major changes have recently occured to the configuration.

Version 9.0.5.xfr

GP Version 5.0.5

Any thoughts?

[vathreya]

Hi Inclusa-Admin,

Can you collect client logs when this happens next and post a snippet (for the exact time stamp) in comments? Can you also let me know since when is this behaviour occurring? 

Regards,

Varun

[inclusa-admin]

This issue started occurring on Monday morning. We had recently updated our domain certificate last week, but did not see any issues up until the past week. The following error is what I am seeing in the logs

(T1388) 04/13/20 11:26:59:346 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_RESOLVING_NAME, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:347 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_NAME_RESOLVED, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:347 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_CONNECTING_TO_SERVER, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:393 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_CONNECTED_TO_SERVER, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:425 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_SECURE_FAILURE, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:425 Info (3359): winhttpObj, dwCertError is:
(T7560) 04/13/20 11:26:59:425 Info (3365): WINHTTP_CALLBACK_STATUS_FLAG_SECURITY_CHANNEL_ERROR
(T7560) 04/13/20 11:26:59:425 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:425 Debug(3419): WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, error=12175, result=5, dwCertificateError=-2147483648
(T7560) 04/13/20 11:26:59:425 Debug(4970): we get cert error, so remove previousCertificate
(T9052) 04/13/20 11:26:59:461 Debug(4930): send alive message now 3
(T9052) 04/13/20 11:26:59:461 Info (2188): winhttpObj, get WINHTTP_CALLBACK_STATUS_REQUEST_ERROR
(T9052) 04/13/20 11:26:59:461 Info (2190): winhttpObj, ERROR_WINHTTP_SECURE_FAILURE set
(T9052) 04/13/20 11:26:59:461 Info (1577): Server cert query failed with error 12019
(T9052) 04/13/20 11:26:59:461 Debug(1489): DC, dump server certificate now

(T11200) 04/13/20 11:26:59:461 Debug( 563): Send command to Pan Service
(T11200) 04/13/20 11:26:59:461 Debug( 590): Command = <request><type>pan_msg_ping</type><result>3</result></request>
(T11200) 04/13/20 11:26:59:461 Debug( 642): PanClient sent successful with 80 bytes
(T9052) 04/13/20 11:26:59:461 Debug(1530): DC, read 2167 of 2167 bytes from file <Removed for security>\AppData\Local\Palo Alto Networks\GlobalProtect\ServerCert.pan.
(T9052) 04/13/20 11:26:59:461 Debug(1395): DC, exportFirstCert

(T9052) 04/13/20 11:26:59:461 Debug(1472): DC, could not find right property id, last error=80092004

(T9052) 04/13/20 11:26:59:461 Error(2218): error = ERROR_WINHTTP_SECURE_FAILURE
(T9052) 04/13/20 11:26:59:461 Debug(4379): do not enforce 1.2, retry it now
(T9052) 04/13/20 11:26:59:461 Info (2105): winhttpObj, SendRequest, m_clientCertName=(null), bIngoreClientCert=0
(T9052) 04/13/20 11:26:59:462 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_RESOLVING_NAME, this=00000206F9DC73E0)
(T9052) 04/13/20 11:26:59:462 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_NAME_RESOLVED, this=00000206F9DC73E0)
(T9052) 04/13/20 11:26:59:462 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_CONNECTING_TO_SERVER, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:493 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_CONNECTED_TO_SERVER, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:530 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_RESOLVING_NAME, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:530 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_NAME_RESOLVED, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:530 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_CONNECTING_TO_SERVER, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:561 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_CONNECTED_TO_SERVER, this=00000206F9DC73E0)
(T9052) 04/13/20 11:26:59:571 Debug(4930): send alive message now 3
(T11200) 04/13/20 11:26:59:571 Debug( 563): Send command to Pan Service
(T11200) 04/13/20 11:26:59:571 Debug( 590): Command = <request><type>pan_msg_ping</type><result>3</result></request>
(T11200) 04/13/20 11:26:59:571 Debug( 642): PanClient sent successful with 80 bytes
(T7560) 04/13/20 11:26:59:597 Info (3346): PanWinhttpCallback(dwInternetStatus=WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, this=00000206F9DC73E0)
(T7560) 04/13/20 11:26:59:597 Debug(3419): WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, error=590615, result=5, dwCertificateError=-2147483648
(T9052) 04/13/20 11:26:59:680 Info (2188): winhttpObj, get WINHTTP_CALLBACK_STATUS_REQUEST_ERROR
(T9052) 04/13/20 11:26:59:680 Info (2190): winhttpObj, ERROR_WINHTTP_SECURE_FAILURE set
(T9052) 04/13/20 11:26:59:680 Info (1577): Server cert query failed with error 12019
(T9052) 04/13/20 11:26:59:680 Debug(1489): DC, dump server certificate now

(T9052) 04/13/20 11:26:59:680 Debug(1530): DC, read 2167 of 2167 bytes from file <Removed for security>\AppData\Local\Palo Alto Networks\GlobalProtect\ServerCert.pan.
(T9052) 04/13/20 11:26:59:680 Debug(1395): DC, exportFirstCert

(T9052) 04/13/20 11:26:59:680 Debug(1472): DC, could not find right property id, last error=80092004

(T9052) 04/13/20 11:26:59:680 Error(2218): error = ERROR_WINHTTP_SECURE_FAILURE
(T9052) 04/13/20 11:26:59:680 Error(4540): winhttpObj, error! ipaddress <Removed for security>

[vathreya]

Hi Inclusa-Admin,

Can you check this reddit post and see if the issue is similar? https://www.reddit.com/r/paloaltonetworks/comments/702d6n/question_err_ssl_protocol_error_globalprot...

If not, please open a TAC case

[inclusa-admin]

Thanks for the information. We are not using a SHA1 cert and I do have a TAC case open at this time. The strange thing is everything was working without issue up until Monday of this week. Rebooting the firewall is a fix, but not the one I like.

[jgibbs]

 Did TAC ever find a solution to this? 

[DSBIII]

Hey Inclusa-Admin.  I think we are facing a similar problem that is intermittent.  Were you able to find a solution?

[MP18]

@DSBIII 

Please look at this link below for recommended version of GP client.

https://live.paloaltonetworks.com/t5/customer-resources/support-pan-os-software-release-guidance/ta-...

I have see many times with GP upgrading the software to recommended version fixes the issues.

Regards