Stack overflow in PanGpHipMp.exe

Reply
Highlighted
L2 Linker

Stack overflow in PanGpHipMp.exe

On my laptop, I'm receiving the following crash notifications in my Windows EventLog about 2-5 times a day.

 

Faulting application name: PanGpHipMp.exe, version: 5.0.8.4, time stamp: 0x5e28f98d
Faulting module name: PanGpHipMp.exe, version: 5.0.8.4, time stamp: 0x5e28f98d
Exception code: 0xc00000fd
Fault offset: 0x00000000000248b7
Faulting process id: 0x1ce0
Faulting application start time: 0x01d62f08de83cc19
Faulting application path: C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHipMp.exe
Faulting module path: C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHipMp.exe
Report Id: 7fa7bd0b-7f4f-45fc-ba7a-b7b8825af55f
Faulting package full name: 
Faulting package-relative application ID:

 

Except that, my GlobalProtect works OK.

The 0xc00000fd is a well-known "stack overflow" exception, e.g. there's a 99.99% chance that there's a bug in "PanGpHipMp.exe", IMHO. Interestingly, I've not found any other error reports, mentioning "0x00000000000248b7".

Not sure what might be the cause. Any idea?


Accepted Solutions
Highlighted
Community Team Member

Hi @i3vi3v ,

 

A PanGpHipMp crash issue has been addressed in GPC-10176 - Fixed in 5.0.9 (also GP 5.1.x is not affected by this issue) :

GlobalProtect App 5.0.9 Addressed Issues 

 

Current preferred release is GP 5.0.10 or GP 5.1.3 :

Support PAN-OS Software Release Guidance 

 

I'd recommend upgrading your GP to one of the recommended versions to rule out you're not hitting this bug GPC-10176.

 

Hope this helps !

-Kiwi.

View solution in original post

Tags (1)

All Replies
Highlighted
L4 Transporter

Hi @i3vi3v ,

 

This will require little more investigation. Is it possible for you to open a TAC case and attach GlobalProtect log bundle. Following can help collecting logs:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaLCAS

 

Thanks,

Nehal

Highlighted
L2 Linker

Hi Nnaik,

Thanks for the suggestion to check that logs. And sorry for such a late reply.
Sadly, I'm unable to open a normal GlobalProtect support ticket. I don't have the codes that are required to created that sort of the account. And, sadly, our IT is not interested in investigating the issue - they only recommend to reimage the system, if I find the issue annoying. Also, they've reinstalled the GlobalProtect via their standard procedure (with some additional cleanup after uninstall) - nothing changed.

I've collected the logs and I can share them if anyone is interested. The "PanGpHipMp.log" seem to be the most relevant, according to it's name. The last few lines are:

 

(T21004) 05/28/20 13:51:52:490 Debug( 77): Opswat is inited.
(T8512) 05/28/20 14:50:08:119 Debug( 118): Found process PanGpHipMp.exe
(T8512) 05/28/20 14:50:08:119 Debug( 54): PanGpHipMp started...
(T8512) 05/28/20 14:50:17:036 Debug(1908): Opswat engine version: 4.3.1087.0
(T8512) 05/28/20 14:50:17:036 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T8512) 05/28/20 14:50:17:036 Debug( 77): Opswat is inited.
(T14672) 05/28/20 14:51:10:739 Debug( 118): Found process PanGpHipMp.exe
(T14672) 05/28/20 14:51:10:739 Debug( 54): PanGpHipMp started...
(T14672) 05/28/20 14:51:18:873 Debug(1908): Opswat engine version: 4.3.1087.0
(T14672) 05/28/20 14:51:18:873 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T14672) 05/28/20 14:51:18:873 Debug( 77): Opswat is inited.
(T8936) 05/28/20 14:52:01:153 Debug( 118): Found process PanGpHipMp.exe
(T8936) 05/28/20 14:52:01:153 Debug( 54): PanGpHipMp started...
(T8936) 05/28/20 14:52:09:926 Debug(1908): Opswat engine version: 4.3.1087.0
(T8936) 05/28/20 14:52:09:926 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T8936) 05/28/20 14:52:09:926 Debug( 77): Opswat is inited.
(T19924) 05/28/20 15:40:51:914 Debug( 118): Found process PanGpHipMp.exe
(T19924) 05/28/20 15:40:51:914 Debug( 54): PanGpHipMp started...
(T19924) 05/28/20 15:41:03:996 Debug(1908): Opswat engine version: 4.3.1087.0
(T19924) 05/28/20 15:41:03:996 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T19924) 05/28/20 15:41:03:996 Debug( 77): Opswat is inited.


And, on the other hand, there's a 0xc00000fd crash events (just like those in my original post above) on 14:50:20, 14:51:21, 14:52:13, 15:41:07.
Quick googling suggests that Opswat is integrated into Global Protect somehow. Well, I even was not aware of this.

There's just one more file, containing "Opswat" substring - the "PanGpHip.log". Interestingly, there are no records around 14:50..14:51 there. For 14:52 the most relevant part, AFAIU, is:

 

(T4656) 05/28/20 13:52:28:914 Debug( 123): All done, deinit opswat
(T9992) 05/28/20 14:52:30:676 Debug( 41): PanGpHip started...
(T9992) 05/28/20 14:52:30:676 Debug( 41): PanGpHip started...
(T9992) 05/28/20 14:52:30:692 Debug( 48): hipObj->m_bIsRoamingProfile: false
(T9992) 05/28/20 14:52:30:698 Debug( 113): Get shared translate length 24
(T9992) 05/28/20 14:52:30:876 Debug( 196): Hip policy is restored from file HipPolicy.dat.
(T9992) 05/28/20 14:52:30:876 Debug( 324): ClearHipCustomCheckInfo(): pHipCustomCheckInfo is NULL.
(T9992) 05/28/20 14:52:30:876 Debug( 86): ClearHipCustomCheckRegKeyInfo(): pHipCustomCheckRegKeyInfo is NULL.
(T9992) 05/28/20 14:52:30:876 Debug( 169): Optional root-ca does not exist
(T9992) 05/28/20 14:52:30:876 Debug( 145): pan_obj_get_value() failed, exclusion 
(T9992) 05/28/20 14:52:30:876 Debug( 262): pan_obj_get_value() failed, exclusion-v4 
(T9992) 05/28/20 14:52:30:876 Debug( 60): initOpswat
(T9992) 05/28/20 14:52:43:426 Debug(1908): Opswat engine version: 4.3.1087.0
(T9992) 05/28/20 14:52:43:426 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T9992) 05/28/20 14:52:43:426 Debug( 81): Opswat is inited.
<...>
(T9992) 05/28/20 14:52:43:590 Debug( 744): Check antimalware category...
(T9992) 05/28/20 14:52:45:098 Debug(1970): Opswat Error(-12): An error when a method call was made on a component that does not implement it. Product: Carbon Black Defense Sensor (Ver: 3.4.0.1052, Vendor: Carbon Black, Inc.), Method: WAAPI_MID_GET_LAST_SCAN_TIME(V3V4), Signature: 491000, Category: 5(ANTIMALWARE), OESIS (V4 ver: 4.3.1087.0, V3V4 ver: 4.3.769.0)
(T9992) 05/28/20 14:52:48:461 Debug( 755): Check firewall category...
(T9992) 05/28/20 14:52:58:608 Debug( 766): Check patch management category...
(T9992) 05/28/20 14:53:07:788 Debug( 777): Check disk encryption category...
(T9992) 05/28/20 14:53:08:262 Debug( 788): Check backup client category...
(T9992) 05/28/20 14:53:08:657 Debug( 799): Check data loss prevention category...
(T9992) 05/28/20 14:53:08:806 Debug( 810): Check antimalware category V4...
(T9992) 05/28/20 14:53:10:050 Debug(1970): Opswat Error(-12): An error when a method call was made on a component that does not implement it. Product: Carbon Black Defense Sensor (Ver: 3.4.0.1052, Vendor: Carbon Black, Inc.), Method: WAAPI_MID_GET_LAST_SCAN_TIME(V4), Signature: 2864, Category: 5(ANTIMALWARE), OESIS (V4 ver: 4.3.1087.0, V3V4 ver: 4.3.769.0)
(T19944) 05/28/20 14:53:13:633 Debug( 150): Hip checking has not been finished within max allowed time 30000ms
(T19944) 05/28/20 14:53:13:633 Debug( 154): Antimalware: 1, Firewall: 1, PatchManagement: 1, DiskEncryption: 1, BackClient: 1, Dlp: 1
(T19944) 05/28/20 14:53:13:633 Debug( 156): Antimalware V4: 0, Firewall V4: 0, PatchManagement V4: 0, DiskEncryption V4: 0, BackClient V4: 0, Dlp V4: 0
(T19944) 05/28/20 14:53:13:633 Debug( 158): Send partial hip report
(T19944) 05/28/20 14:53:13:638 Debug( 583): Machine's device id is 5ad863ad-77a3-4e3a-8e88-74bebdc3ded7
(T19944) 05/28/20 14:53:13:705 Debug( 916): Updated hip category report file HIP_AV_Report.dat
(T19944) 05/28/20 14:53:13:729 Debug( 916): Updated hip category report file HIP_AS_Report.dat
(T19944) 05/28/20 14:53:13:747 Debug( 916): Updated hip category report file HIP_BC_Report.dat
(T19944) 05/28/20 14:53:13:756 Debug( 916): Updated hip category report file HIP_DE_Report.dat
(T19944) 05/28/20 14:53:13:785 Debug( 916): Updated hip category report file HIP_FW_Report.dat
(T19944) 05/28/20 14:53:13:785 Debug( 567): pan_read_text_from_file(): File does not exist. File: C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpMPR.dat
(T19944) 05/28/20 14:53:13:797 Debug( 916): Updated hip category report file HIP_PM_Report.dat
(T19944) 05/28/20 14:53:13:818 Debug( 916): Updated hip category report file HIP_DLP_Report.dat

There's also something similar around 15:41.
It's interesting that both mentioned "Error(-12)" messages were logged after the crash.

Well, I must admit that these logs do not look very promising to me...

 

Highlighted
Community Team Member

Hi @i3vi3v ,

 

A PanGpHipMp crash issue has been addressed in GPC-10176 - Fixed in 5.0.9 (also GP 5.1.x is not affected by this issue) :

GlobalProtect App 5.0.9 Addressed Issues 

 

Current preferred release is GP 5.0.10 or GP 5.1.3 :

Support PAN-OS Software Release Guidance 

 

I'd recommend upgrading your GP to one of the recommended versions to rule out you're not hitting this bug GPC-10176.

 

Hope this helps !

-Kiwi.

View solution in original post

Tags (1)
Highlighted
L2 Linker

Hi Kiwi,

I've upgraded to 5.0.9 about a week ago (it is now the standard version in our corporate environment), and I can confirm the issue is gone since then.

Thanks for fixing!

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!