- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
02-16-2022 07:33 AM - edited 02-16-2022 07:35 AM
Hi,
We are having a strange issue in our FW. User in VPN-SSL reported the stop working. The issue doesnt have any pattern. Random users, random time-range.
The issue is solved when the customer force to reconnect the VPN or force pass the HIP check in GPclient.
This is what we see in FW monitor logs. The FW stops identifying user and jump the rule.
I add the monitor traffic:
what log file can give me info (useriid, authd)
02-18-2022 12:23 AM
Hi @BigPalo
- What OS version are you using?
- Have you checked the User-ID logs in the GUI filtering for one specific user? What is the timeout, data source? Do you see any logout events for that user?
- Do you see any process crash dump files - "> show system files"
- Try to correlate GlobalProtect logs and Traffic logs - is there any pattern in the time between last GP log for given user that the first traffic log without user-id?
02-18-2022 01:02 AM
In addition to the questions that @aleksandar.astardzhiev has already asked, what version of the GlobalProtect agent are you using? Is everyone running the same version?
02-18-2022 04:06 AM
Hi @BigPalo,
I shouldn't have reply to this thread....I just receive exact same complate for one of our users.
From my troubleshooting it seems there the problem is in the GlobalPortect agent.
- Checking ip-to-user mapping I can see that there is no record for the IP address of the affected user. Even that he is still connected to GlobalProtect
- Checking GlobalPortect logs it seems he still perform portal config refresh, because I can see his initial login and consequent portal connects every 2 hours.
- Checking User-ID logs I can see that entry was created with timestamp when user initialy connected to GlobalPortect, with timeout of 3 hours (10800sec) - at the moment I am not sure if this value depends on my config or it is specific for GP connected users.
- There is no other user-id log for that user-to-ip mapping after that.
- Looking at traffic logs I can see that exactly after 3hours after the last user-id log, firewall has lost mapping - because the mapping timeout has expired and there was no event that will refresh that timer.
To be honest I don't know how user-to-ip mapping entries for GP users are refreshed. Probably @BPry can correct me, but my guess is with the periodic HIP report re-submission for connected GP users.
- Checking HIP match logs and User-ID logs for non-affected user (myself) I can see that everytime threre is HIP match log there is user-id log, which is every hour.
- Checking HIP match logs and User-ID logs for affected user - I can see that his last HIP log was from his initial GP connect (exact same time user mapping is created) and here are no periodical logs for hip re-submission, like it should have every hour.
So it for me all boils down to - Why user is not submitting HIP report periodically if he is still connected and active in GP.
- Our GlobalProtect agent is 5.2.9
- Affected user is running MacOS 12.2 (I don't have information for other affected users yet)
- Non-affected user is running Windows
I will try to gather GP agent logs from affected user and I would suggest you to do the same.
02-18-2022 07:20 AM
Looking at GlobalPortect logs for the affected user I notice something strange:
- It seems HIP report is actually generated every hour
- But GlobalProtect "decides" not to submit it to the gateway, event that it is connected
From PanGPS.log I can see following lines repeating every hour, until the user refreshed vpn connection
02/18/2022 09:25:53:922 Debug(6441): HIP report v4 md5 digest is 5ce82f8f45af2ab3c4eee3e1f4ba329a
02/18/2022 09:25:53:922 Debug(6469): HipReportThread: network type is external network.
02/18/2022 09:25:53:923 Debug(1675): Send hip report to gateway gp-gva.uefa.ch
02/18/2022 09:25:53:923 Debug(4811): Entering SendHipReportToGateway(). Gateway: <gp-gateway-fqdn>
02/18/2022 09:25:53:923 Debug(4819): has not logged into gateway <gp-gateway-fqdn>. Skip sending hip report to this gateway.
02/18/2022 09:25:53:923 Debug(1681): SendHipReportToGateway gp-gva.uefa.ch returns FALSE.
@BigPalo I am intersted if your logs are showing something similar.
02-18-2022 07:54 AM
Interesting. From your logs it looks as though the agent isn't even registered to the gateway, or at least the agent/gateway thinks it's not registered to the gateway. On the firewalls logs do you see the agent possibly hitting the inactivity timeout? I've seen issues where people are resetting the HIP report due to URL filtering on the gateway security entries getting it caught up since it uses the gateway IP, and that can cause the inactivity timer to trigger and have the agent get "booted" from the gateway.
02-18-2022 08:49 AM - edited 02-18-2022 08:50 AM
Hey @BPry
I was also thinking for something similar, but it doesn't seems to have valid reason why gp agent to "think" it is not connected.
Below logs show (filtered by username):
- Initial user login
- Portal config refresh (exactly 2 hours after login)
- GP session expiration due to inactivity (4 hours of "inactivity" after initial login)
- User triggresh "refresh connection" from GP agent GUI
Problem has started at 11:53 - one hour before GP inactivity session timer to timeout
User-ID mapping shows:
- Initial GP login
- HIP report submitted 30mins laters
- GP session expiration due to inactivity - one hour after the problem has occured
- User triggered "refresh connnection"
(you may diregard logs from user-id agent, it is redistribution loop)
From PanGPS logs:
- User has connected to GP gateway
02/18/2022 08:23:39:712 Debug( 733): ipsec connect(<gateway-ip>) succeed
02/18/2022 08:23:39:712 Debug(11199): VPN tunnel is connected.
02/18/2022 08:23:39:712 Debug(11203): Enable life time and create life time thread.
02/18/2022 08:23:39:712 Debug(7064): --Set state to Connected
02/18/2022 08:23:39:897 Debug(2615): Tunnel is created with the gateway <gp-gateway-fqdn>
- HIP report is submitted after login
02/18/2022 08:23:40:373 Info (5155): sent HIP report to <gp-gateway-fqdn>.
02/18/2022 08:23:40:374 Debug(5183): Response status of HIP report is success, <gp-gateway-fqdn>
02/18/2022 08:23:40:374 Debug(5185): Hip report returns success.
02/18/2022 08:23:40:374 Info (4949): Got hip notification from gateway <gp-gateway-fqdn>.
02/18/2022 08:23:40:374 Debug(4957): Hip notification is empty in the HIP report check response from gateway <gp-gateway-fqdn>
- After 30min agent re-submit HIP
02/18/2022 08:53:47:882 Debug(4866): Time to send hip report to gateway <gp-gateway-fqdn>
02/18/2022 08:53:47:882 Debug(2984): Gateway: <gp-gateway-fqdn>, client IP: <user-local-ip>
02/18/2022 08:53:47:882 Debug(4883): Hip report head to gateway <gp-gateway-fqdn>is
- One our later after login, GP has prepared the HIP report, but didn't send it
02/18/2022 09:25:53:922 Debug(6469): HipReportThread: network type is external network.
02/18/2022 09:25:53:923 Debug(1675): Send hip report to gateway <gp-gateway-fqdn>
02/18/2022 09:25:53:923 Debug(4811): Entering SendHipReportToGateway(). Gateway: <gp-gateway-fqdn>
02/18/2022 09:25:53:923 Debug(4819): has not logged into gateway <gp-gateway-fqdn>. Skip sending hip report to this gateway.
02/18/2022 09:25:53:923 Debug(1681): SendHipReportToGateway <gp-gateway-fqdn> returns FALSE.
I could be missing something, but I don't see any event in GP logs between initial connect and failed HIP, that could indicate that GP is not connected to gateway.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!