GlobalProtect client frequently disconnects with "Received a virtual interface packet with fragment"

[millerh1]

I am experiencing frequent disconnect events which typically last 30-90 seconds and occur about 2-3x an hour during the work day. When a disconnect event happens, all connectivity is lost from the user's point of view, however the GlobalProtect client remains "connected". When these events occur, the PanGPS.log file shows the following:

(T10852)Debug(1564): 01/11/21 17:33:00:578 Received a virtual interface packet with fragment 0x2000
(T10852)Debug(1564): 01/11/21 17:33:00:578 Received a virtual interface packet with fragment 0xAC
(T10852)Debug(1564): 01/11/21 17:33:01:779 Received a virtual interface packet with fragment 0x2000
(T10852)Debug(1564): 01/11/21 17:33:01:779 Received a virtual interface packet with fragment 0xAC
...

These lines are repeated until connectivity is restored. I am on GlobalProtect client 5.2.4-21.

Does anyone have an idea what might be causing this?

[kiwi]

Hi @millerh1 ,

A possible reason for this could be that a lot of fragmented packets are coming and as a result the packet queue is getting filled up as it has to wait for reassembling.  The filled up queue might be causing the connection delays.

You could check the minumum MTU of the PANGP virtual adapter to avoid IP fragmentation caused by the ISP or any device in the transit.  If you have identified the minimum MTU you can change the virtual adapter MTU following this article:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PPbZCAW

The default MTU size in a GP adapter is 1400 bytes.

To identify the minimum MTU size you can send a large PING size to 8.8.8.8 (for example) and reduce it until you get a reply as seen in the example below :

ping -l 1373 -f 8.8.8.8

Pinging 8.8.8.8 with 1373 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), 

ping -l 1372 -f 8.8.8.8
Pinging 8.8.8.8 with 1372 bytes of data: 
Reply from 8.8.8.8: bytes=68 (sent 1372) time=273ms TTL=117
Reply from 8.8.8.8: bytes=68 (sent 1372) time=274ms TTL=117
Reply from 8.8.8.8: bytes=68 (sent 1372) time=274ms TTL=117
Reply from 8.8.8.8: bytes=68 (sent 1372) time=274ms TTL=117. <<<<---------------That's how you'll get know the minimum size of MTU(1372 is the minimum mtu size in this example, verify like this only at your end).

Note that reducing the MTU size might cause other issues ... I would recommend gathering all your logs (Tech support file + GP debug logs) and reach out to support for further analysis of the logs/traffic.

Cheers,

-Kiwi.

[millerh1]

Thank you so much for the advice!

I have tried the MTU check:

C:\Users\mille>ping -l 1373 -f 8.8.8.8

Pinging 8.8.8.8 with 1373 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Users\mille>ping -l 1372 -f 8.8.8.8

Pinging 8.8.8.8 with 1372 bytes of data:
Reply from 8.8.8.8: bytes=68 (sent 1372) time=131ms TTL=117
Reply from 8.8.8.8: bytes=68 (sent 1372) time=123ms TTL=117
Reply from 8.8.8.8: bytes=68 (sent 1372) time=131ms TTL=117
Reply from 8.8.8.8: bytes=68 (sent 1372) time=125ms TTL=117

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 123ms, Maximum = 131ms, Average = 127ms

It appears that my minimum MTU may also be 1372, just like you showed. 

It sounds like you are advising that I try lowering the MTU on the PanGP adapter from 1400 (the default) to something below 1372. I lowered it to 1300 by identifying the index of the GP adapter (idx 18 on my computer) and then setting the mtu for that adapter to 1300. I did this for ipv4 (shown) and ipv6.

C:\Windows\system32>netsh interface ipv4 show interfaces

Idx     Met         MTU          State                Name
---  ----------  ----------  ------------  ---------------------------
  1          75  4294967295  connected     Loopback Pseudo-Interface 1
 11          70        1500  disconnected  Wi-Fi
 22          25        1500  connected     Ethernet
 17          25        1500  disconnected  Local Area Connection* 1
  7          25        1500  disconnected  Local Area Connection* 2
 18           1        1400  connected     Ethernet 2
 23        5000        1500  connected     vEthernet (WSL)


C:\Windows\system32>netsh interface ipv4 set interface 18 mtu=1300
Ok.

However, the disconnect events still persist with the same debug message:

(T30932)Debug(1564): 01/12/21 08:33:05:755 Received a virtual interface packet with fragment 0x2000
(T30932)Debug(1564): 01/12/21 08:33:05:761 Received a virtual interface packet with fragment 0xA0
(T30932)Debug(1564): 01/12/21 08:33:10:557 Received a virtual interface packet with fragment 0x2000
(T30932)Debug(1564): 01/12/21 08:33:10:557 Received a virtual interface packet with fragment 0xA0
(T30932)Debug(1564): 01/12/21 08:33:20:121 Received a virtual interface packet with fragment 0x2000
(T30932)Debug(1564): 01/12/21 08:33:20:121 Received a virtual interface packet with fragment 0xA0

I will do my best to contact support -- I do not know if our institute has a support account, however. 

Do you have any other suggestions regarding what might be causing this? 

Thank you for your time,

Henry

[RMikalauskas]

Hello,

Were you able to involve PAN TAC into this?

[MP18]

@millerh1 

Any update from PA on this?

Regards

[millerh1]

@MP18and @RMikalauskas  Yes we are currently in contact with them. They are still reviewing our logs and we are looking at different possible causes. 

[thomaswiesner]

I have exactly the same issue like you.

PA-3260 with Pan-OS 9.1.7 and GP 5.2.5. on Windows 10.

The error only occurs on a IPv6 connection, IPv4 is fine.

MTU is 1388 for and still got the problems.

Did you get a solution from TAC @millerh1 ?

[millerh1]

Still looking into the problem. They asked me to get a new modem/router and see if the problem goes away -- currently testing that now.

[thomaswiesner]

Just as und Information:

we see it most times when using Citrix. 

Do you also have some kind of special application where it happens most?

[millerh1]

No -- but it does happen for me at certain times of day. Particularly around 9am, 12pm, and 4-5pm. 

Anyways, the new modem and router seems to have fixed it! Still would like to know why this is happening since I don't think "buy a new modem/router" is going to be great solution for most people.

[millerh1 · Accepted solution]

Final verdict after a week:

It appears that getting a different modem and router fixed the problem...

Customer support still doesn't know what caused the issue and they are not interested in investigating further as far as I can tell... The equipment I had before (a 2017 Arris Surfboard Gateway) worked fine except when running GlobalProtect.

If you are having this problem, please contact customer support and make sure they are aware that this issue wasn't isolated to one customer.

"Buy a new modem/router" should not be considered a sufficient long-term solution.

[RMikalauskas]

In our case there was a 3rd party software firewall on the endpoint that dropped incoming packets.

[EugeneSourman]

We are having the same issue. Our users drop connection randomly and the logs show fragmented packets. We are also on PrismaAccess (Panorama Managed). We have replaced switches and APs with a hope that it would resolve the issue but it is still there. The only other thing left is the ISPs router.