topic Re: All incoming TCP connections blocked for 5 minutes at random times. in General Topics

All incoming TCP connections blocked for 5 minutes at random times.

sharam — Wed, 01 Jul 2020 14:46:37 GMT

At random times all TCP connections from the Internet are blocked (all ports and all IPs) for incoming traffic only. Websites, mail servers etc are not accessible from the Internet. UDP and ICMP are not affected. What could be causing this?

Re: All incoming TCP connections blocked for 5 minutes at random times.

BPry — Wed, 01 Jul 2020 18:29:24 GMT

Sounds like a DoS profile that is setup incorrectly and is blocking the destination address (your public IP) or vulnerability protection profile with the same misconfiguration. The default block-ip timeout is 5 minutes, which lines up with what you are running into.

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Wed, 01 Jul 2020 20:20:15 GMT

There are no entries under Policy -> DoS Protection.

Is there another location I can look?

Re: All incoming TCP connections blocked for 5 minutes at random times.

BPry — Wed, 01 Jul 2020 20:27:59 GMT

@sharam,

If you had any that's where they would be. What does your Vulnerability Protection profile look like for the security rulebase entries allowing this traffic inbound? That would be the next thing I would look at.

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 15:03:22 GMT

What I am seeing in logs related to the outage is:

But I am not able to see which rulebase this it related to.

Re: All incoming TCP connections blocked for 5 minutes at random times.

Brandon_Wertz — Mon, 06 Jul 2020 16:24:55 GMT

@sharam wrote:

What I am seeing in logs related to the outage is:

But I am not able to see which rulebase this it related to.

You're showing the detailed log entry. (You've clicked the magnifying glass next to an entry in the threat log) To better help point out why this is happening we need to see the main entry. It looks like this is happening because of a VP profile which is attached to a security policy rule.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 18:32:47 GMT

zone protection

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 18:52:16 GMT

Thank you for your response @Brandon_Wertz . This is the main entry of an event that just happened today:

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 19:07:45 GMT

Can you please be more specific? Zone Pro is the first place we looked and changed various config to no effect.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 19:28:04 GMT

Did you look under Objects -> Security Profiles -> Dos Protection? Th default block duration is 5 minutes..

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 19:35:26 GMT

@fhewiufhwefhwe there are no entries under Objects -> Security Profiles -> Dos Protection, not even default.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 19:37:58 GMT

Do you have source address exclusions under Reconnaissance Protection? You may want to include highly trusted items on your allow list there, such as internal DNS forwarders if you have them.

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 19:46:34 GMT

We don't have this problem with UDP which is what DNS uses.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 19:54:50 GMT

DNS does use UDP, but can also use TCP over port 53 as well. Any way if you have site-to-site vpn or something similar, you may want some exclusions to Dos / Zone protections policies.

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 20:18:39 GMT

Thanks @fhewiufhwefhwe but the problem we are facing is that when the TCP outage occurs our thousands of user who are scattered across the Internet world are not able to reach our websites or mail servers. We don't want the firewall to block the whole Internet.

I should also point out if the subject is not clear that the problem is only with incoming TCP traffic. Outgoing traffic is not affected.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 20:28:15 GMT

Have you tried increasing the thresholds under SYN Flood Protection then? How high is your firewall on resource utilization? I'm wondering if you might be able to use SYN Cookies instead of Random Early Drop action? Note that I have not tried this myself.

I'm also thinking that you might want to setup SNMP monitoring the DoS counters on the firewall. I recently tried to configure this, but have not had success yet.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 20:43:07 GMT

This article might help. I've been meaning to read it myself and figure out how to setup snmp monitoring for DoS.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClOKCA0

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 20:53:42 GMT

I have increased the threshold from 100 to 500 under Reconnaissance per your suggestion. Will see if that makes a difference.

Re: All incoming TCP connections blocked for 5 minutes at random times.

fhewiufhwefhwe — Mon, 06 Jul 2020 20:58:53 GMT

I did not mean the threshold under Reconnaissance Protection. I mean the threshold under Flood Protection. The defaults are 10,000 fro Alarm Rate and Activate, and 40,000 for maximum connections/sec.

Re: All incoming TCP connections blocked for 5 minutes at random times.

sharam — Mon, 06 Jul 2020 21:13:17 GMT

Those are already pretty high. Alarm and active reates are 59,000. Maximum connections at 100,000.