User's traffic not hitting correct security rule.

[AICOFINDIA]

We're running into an issue where a rule that is meant to update anti-virus protection on port 443 is slipping through and being caught by a lower rule which denies any application and service. (Hardware: PA-5050, OS version : 8.1.6).

As far as the security rule is concerned, we have mentioned FQDNs as the destinations (instead of IPs & URLs) while source has specific IPs.

[aleksandar.astardzhiev]

Hi @AICOFINDIA ,

Compare the log  you see from the denied traffic with your "AV update" rule:

- First check what FW has detected as application when matched  your deny rule - is it matching the application in your "AV update"?

- What about service port? Are you using "application-default" for the allow rule?

- Check if the FQDN for the update is not doing any "dns loadbalancing" and returning different IP address on each request (run multiple nslookups from your pc - is the reply change requently)

Most common reasons in my experiance are:

- In your allow rule you are allowing application ssl, while the FW is detecting different application

- Clients/users are resolving the av update FQDN to differnt IP from what the FW is resolving the FQDN. When using FQDN objects in the policy, FW will run DNS queries for the provided FQDN and put the first N IPs from the dns reply (not sure what was the limit if the dns reply multiple ips for single fqdn) and put them in the rule. FW will cache the dns response for 30mins (by default). If the endpoint and FW are using different DNS servers the may resolve the FQDN to different IP addresses, or the FQDN resolve to differn IP on each request.

[AICOFINDIA]

Hi Alexander,

Here are my findings for further reference :-

- First check what FW has detected as application when matched  your deny rule - is it matching the application in your "AV update"?

The traffic log has detected "not-applicable" as application.

- What about service port? Are you using "application-default" for the allow rule?

We have mentioned "ANY" for the allow rule.

- Check if the FQDN for the update is not doing any "dns loadbalancing" and returning different IP address on each request (run multiple nslookups from your pc - is the reply change requently)

It's a static IP.

NOTE:-

1.) The deny rule has been created manually to block everything : any-to-any (IP, service, application, port, zone) with security profile enabled in Actions > Profile Setting. 

2.) This connectivity is working perfectly fine on our other FW deployment (at other Data Center) that is running an identical hardware, OS version, FW policy as this one (except some differences in Actions > Profile Setting).

3.) Last but not least, it has worked perfectly fine in both locations until a few days back when it was reported not updating the endpoint devices by the customer.

[OtakarKlier]

Hello,

Also with the above solutions, I say open a support case. The reason for this is that yesterday I also noticed traffic randomly hitting incorrect policies and getting denied. I have yet to have a permanent fix/ reason from support.

Regards,

[Remo]

Hi @AICOFINDIA 

Did you check if the firewall is able to resolve the fqdn? Mybe for some reason this firewall is not able to resolve this fqdn, which would be an explanation for your described problem.

If it is a static IP, did you try to test if the policy works properly with an IP object instead of fqdn?

[AICOFINDIA]

Hi Vsys_remo,

It's working now perfectly well & the firewall is able to resolve the FQDN too. This brings to mind an exact occurrence 3 months back when the issue cropped up all of a sudden & got resolved without changing the firewall settings. As far as our observation is concerned, it has been an intermittent issue for the last 6 months or so. Sometimes, the firewall works well while at other times, it does not. I am still confused what is causing the issue to occur out of nowhere.

[AICOFINDIA]

Hi there,

No anomalous behavior has been observed for the last couple of days & that the concerned traffic is up & running fine (NOTE:- no config changes were made). Can anyone please explain why on earth this all happened? Can we expect any such behavior in the future too?

[OtakarKlier]

Hello All,

Just an update from my similar issue. Have been working with support with no clear resolution. They did mention that there was 1 case in the system where the customer upgraded their firewall and the issue was resolved. I will be upgrading ours this saturday and will advise if the issue is corrected.

Regards,

[AICOFINDIA]

Hi OtakarKlier,

You are requested to share the update afterward. What OS version is your firewall running currently & what OS version have you planned it to upgrade to?

[OtakarKlier]

Hello,

We are on 9.0.12 and going to 9.0.13. Will update next week after the upgrade.

Regards,

[samkolars]

I've seen BugID in release notes that suggest there's been fixes to an issue with FQDN objects resolving and the IP's associated with them not making it into running configuration due to there being uppercase letters in the FQDN object.  I don't recall if it was in reference to the object name having uppercase letters in it or the FQDN value within the object having the uppercase letters.  I've had the same issues in the past but I've kept this in mind when creating and using FQDN objects in policy since reading this.