Stack overflow in PanGpHipMp.exe

[i3vi3v]

On my laptop, I'm receiving the following crash notifications in my Windows EventLog about 2-5 times a day.

Faulting application name: PanGpHipMp.exe, version: 5.0.8.4, time stamp: 0x5e28f98d
Faulting module name: PanGpHipMp.exe, version: 5.0.8.4, time stamp: 0x5e28f98d
Exception code: 0xc00000fd
Fault offset: 0x00000000000248b7
Faulting process id: 0x1ce0
Faulting application start time: 0x01d62f08de83cc19
Faulting application path: C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHipMp.exe
Faulting module path: C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpHipMp.exe
Report Id: 7fa7bd0b-7f4f-45fc-ba7a-b7b8825af55f
Faulting package full name: 
Faulting package-relative application ID:

Except that, my GlobalProtect works OK.

The 0xc00000fd is a well-known "stack overflow" exception, e.g. there's a 99.99% chance that there's a bug in "PanGpHipMp.exe", IMHO. Interestingly, I've not found any other error reports, mentioning "0x00000000000248b7".

Not sure what might be the cause. Any idea?

[nnaik]

Hi @i3vi3v ,

This will require little more investigation. Is it possible for you to open a TAC case and attach GlobalProtect log bundle. Following can help collecting logs:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaLCAS

Thanks,

Nehal

[i3vi3v]

Hi Nnaik,

Thanks for the suggestion to check that logs. And sorry for such a late reply.
Sadly, I'm unable to open a normal GlobalProtect support ticket. I don't have the codes that are required to created that sort of the account. And, sadly, our IT is not interested in investigating the issue - they only recommend to reimage the system, if I find the issue annoying. Also, they've reinstalled the GlobalProtect via their standard procedure (with some additional cleanup after uninstall) - nothing changed.

I've collected the logs and I can share them if anyone is interested. The "PanGpHipMp.log" seem to be the most relevant, according to it's name. The last few lines are:

(T21004) 05/28/20 13:51:52:490 Debug( 77): Opswat is inited.
(T8512) 05/28/20 14:50:08:119 Debug( 118): Found process PanGpHipMp.exe
(T8512) 05/28/20 14:50:08:119 Debug( 54): PanGpHipMp started...
(T8512) 05/28/20 14:50:17:036 Debug(1908): Opswat engine version: 4.3.1087.0
(T8512) 05/28/20 14:50:17:036 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T8512) 05/28/20 14:50:17:036 Debug( 77): Opswat is inited.
(T14672) 05/28/20 14:51:10:739 Debug( 118): Found process PanGpHipMp.exe
(T14672) 05/28/20 14:51:10:739 Debug( 54): PanGpHipMp started...
(T14672) 05/28/20 14:51:18:873 Debug(1908): Opswat engine version: 4.3.1087.0
(T14672) 05/28/20 14:51:18:873 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T14672) 05/28/20 14:51:18:873 Debug( 77): Opswat is inited.
(T8936) 05/28/20 14:52:01:153 Debug( 118): Found process PanGpHipMp.exe
(T8936) 05/28/20 14:52:01:153 Debug( 54): PanGpHipMp started...
(T8936) 05/28/20 14:52:09:926 Debug(1908): Opswat engine version: 4.3.1087.0
(T8936) 05/28/20 14:52:09:926 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T8936) 05/28/20 14:52:09:926 Debug( 77): Opswat is inited.
(T19924) 05/28/20 15:40:51:914 Debug( 118): Found process PanGpHipMp.exe
(T19924) 05/28/20 15:40:51:914 Debug( 54): PanGpHipMp started...
(T19924) 05/28/20 15:41:03:996 Debug(1908): Opswat engine version: 4.3.1087.0
(T19924) 05/28/20 15:41:03:996 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T19924) 05/28/20 15:41:03:996 Debug( 77): Opswat is inited.

And, on the other hand, there's a 0xc00000fd crash events (just like those in my original post above) on 14:50:20, 14:51:21, 14:52:13, 15:41:07.
Quick googling suggests that Opswat is integrated into Global Protect somehow. Well, I even was not aware of this.

There's just one more file, containing "Opswat" substring - the "PanGpHip.log". Interestingly, there are no records around 14:50..14:51 there. For 14:52 the most relevant part, AFAIU, is:

(T4656) 05/28/20 13:52:28:914 Debug( 123): All done, deinit opswat
(T9992) 05/28/20 14:52:30:676 Debug( 41): PanGpHip started...
(T9992) 05/28/20 14:52:30:676 Debug( 41): PanGpHip started...
(T9992) 05/28/20 14:52:30:692 Debug( 48): hipObj->m_bIsRoamingProfile: false
(T9992) 05/28/20 14:52:30:698 Debug( 113): Get shared translate length 24
(T9992) 05/28/20 14:52:30:876 Debug( 196): Hip policy is restored from file HipPolicy.dat.
(T9992) 05/28/20 14:52:30:876 Debug( 324): ClearHipCustomCheckInfo(): pHipCustomCheckInfo is NULL.
(T9992) 05/28/20 14:52:30:876 Debug( 86): ClearHipCustomCheckRegKeyInfo(): pHipCustomCheckRegKeyInfo is NULL.
(T9992) 05/28/20 14:52:30:876 Debug( 169): Optional root-ca does not exist
(T9992) 05/28/20 14:52:30:876 Debug( 145): pan_obj_get_value() failed, exclusion 
(T9992) 05/28/20 14:52:30:876 Debug( 262): pan_obj_get_value() failed, exclusion-v4 
(T9992) 05/28/20 14:52:30:876 Debug( 60): initOpswat
(T9992) 05/28/20 14:52:43:426 Debug(1908): Opswat engine version: 4.3.1087.0
(T9992) 05/28/20 14:52:43:426 Debug(1913): Opswat V3V4 adaper version: 4.3.769.0
(T9992) 05/28/20 14:52:43:426 Debug( 81): Opswat is inited.
<...>
(T9992) 05/28/20 14:52:43:590 Debug( 744): Check antimalware category...
(T9992) 05/28/20 14:52:45:098 Debug(1970): Opswat Error(-12): An error when a method call was made on a component that does not implement it. Product: Carbon Black Defense Sensor (Ver: 3.4.0.1052, Vendor: Carbon Black, Inc.), Method: WAAPI_MID_GET_LAST_SCAN_TIME(V3V4), Signature: 491000, Category: 5(ANTIMALWARE), OESIS (V4 ver: 4.3.1087.0, V3V4 ver: 4.3.769.0)
(T9992) 05/28/20 14:52:48:461 Debug( 755): Check firewall category...
(T9992) 05/28/20 14:52:58:608 Debug( 766): Check patch management category...
(T9992) 05/28/20 14:53:07:788 Debug( 777): Check disk encryption category...
(T9992) 05/28/20 14:53:08:262 Debug( 788): Check backup client category...
(T9992) 05/28/20 14:53:08:657 Debug( 799): Check data loss prevention category...
(T9992) 05/28/20 14:53:08:806 Debug( 810): Check antimalware category V4...
(T9992) 05/28/20 14:53:10:050 Debug(1970): Opswat Error(-12): An error when a method call was made on a component that does not implement it. Product: Carbon Black Defense Sensor (Ver: 3.4.0.1052, Vendor: Carbon Black, Inc.), Method: WAAPI_MID_GET_LAST_SCAN_TIME(V4), Signature: 2864, Category: 5(ANTIMALWARE), OESIS (V4 ver: 4.3.1087.0, V3V4 ver: 4.3.769.0)
(T19944) 05/28/20 14:53:13:633 Debug( 150): Hip checking has not been finished within max allowed time 30000ms
(T19944) 05/28/20 14:53:13:633 Debug( 154): Antimalware: 1, Firewall: 1, PatchManagement: 1, DiskEncryption: 1, BackClient: 1, Dlp: 1
(T19944) 05/28/20 14:53:13:633 Debug( 156): Antimalware V4: 0, Firewall V4: 0, PatchManagement V4: 0, DiskEncryption V4: 0, BackClient V4: 0, Dlp V4: 0
(T19944) 05/28/20 14:53:13:633 Debug( 158): Send partial hip report
(T19944) 05/28/20 14:53:13:638 Debug( 583): Machine's device id is 5ad863ad-77a3-4e3a-8e88-74bebdc3ded7
(T19944) 05/28/20 14:53:13:705 Debug( 916): Updated hip category report file HIP_AV_Report.dat
(T19944) 05/28/20 14:53:13:729 Debug( 916): Updated hip category report file HIP_AS_Report.dat
(T19944) 05/28/20 14:53:13:747 Debug( 916): Updated hip category report file HIP_BC_Report.dat
(T19944) 05/28/20 14:53:13:756 Debug( 916): Updated hip category report file HIP_DE_Report.dat
(T19944) 05/28/20 14:53:13:785 Debug( 916): Updated hip category report file HIP_FW_Report.dat
(T19944) 05/28/20 14:53:13:785 Debug( 567): pan_read_text_from_file(): File does not exist. File: C:\Program Files\Palo Alto Networks\GlobalProtect\PanGpMPR.dat
(T19944) 05/28/20 14:53:13:797 Debug( 916): Updated hip category report file HIP_PM_Report.dat
(T19944) 05/28/20 14:53:13:818 Debug( 916): Updated hip category report file HIP_DLP_Report.dat

There's also something similar around 15:41.
It's interesting that both mentioned "Error(-12)" messages were logged after the crash.

Well, I must admit that these logs do not look very promising to me...

[kiwi · Accepted solution]

Hi @i3vi3v ,

A PanGpHipMp crash issue has been addressed in GPC-10176 - Fixed in 5.0.9 (also GP 5.1.x is not affected by this issue) :

GlobalProtect App 5.0.9 Addressed Issues 

Current preferred release is GP 5.0.10 or GP 5.1.3 :

Support PAN-OS Software Release Guidance 

I'd recommend upgrading your GP to one of the recommended versions to rule out you're not hitting this bug GPC-10176.

Hope this helps !

-Kiwi.

[i3vi3v]

Hi Kiwi,

I've upgraded to 5.0.9 about a week ago (it is now the standard version in our corporate environment), and I can confirm the issue is gone since then.

Thanks for fixing!