High loads by scanner

[PATRICKKASIE]

Good afternoon. At 11:57:26 (9:57:26 GMT), there was a log entry which said this company was scanning our VPS. This made it unresponsive with extremely high load for us until I had restarted the HTTPD service. The traffic was also coming from a lot of different IP addresses all by Microsoft. Is this normal?

Log entry:

198.235.24.150 - - [14/Jun/2022:11:57:26 +0200] "GET / HTTP/1.1" 200 275 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com"

[hpatel11]

Hey yes that is totally normal, as soon as you put any public IP online you will get scanned by different scanner not just Palo Alto. I would suggest you to secure your external facing ip by adding strict security profile group to block or reset it at firewall.

To prevent it further you can add deny rule on top with EDL You can also add your custom EDL with this to add one off IP. 
Palo Alto Networks - Known malicious IP addresses
Palo Alto Networks - Bulletproof IP addresses
Palo Alto Networks - High risk IP addresses

[PATRICKKASIE]

Thank you for your response. Unfortunately, I'm not sure if I can do that, because we're not part of the Palo Alto's network. That's the thing. We're renting our VPS-es at a webhosting company called TransIP, and we manage the servers ourselves. I'll probably have to ask with them in that case, but I'd still like to know if there is a way to prevent situations like these, where I'm running up to a forum for a company I've never heard of until now, and asking help for this topic. Is there any way to prevent this situation in the future?

[hpatel11]

In that case your best bet is to send email to  scaninfo@paloaltonetworks.com with your IP subnet to exclude from scan. 

[Adrian_Jensen]

How do you know that PaloAlto made your website unresponsive? You have a single log of a request of your index page, something hundreds or thousands of other bots are doing every day, and the PaloAlto bot is explicitly telling you who/what it is in the client header. It is part of PaloAlto's website categorization for threats/malware prevention and typically happens about once a day from one of a handful of PaloAlto IP blocks. It is annoying, but nothing about the request should slow down your website.

I would be looking at those hundreds of other requests and what they were access/trying to PUT.

[PATRICKKASIE]

Yeah you're probably right, it might be a coincidence that this happened as the high load had just commenced. I'm not a pro system administrator and I have to learn a lot.

The access_log doesn't show a whole lot in that regard. Not really a whole lot of requests are written down on it at that time. domain.com/server-status really showed a lot of connections with a lot of requests at the time of looking, all coming from different IP addresses and all were looking up different domains on the VPS.

[Adrian_Jensen]

My own personal server/website gets hit about once a day by the PaloAlto website scanner. Always a single request for the index page. For the last couple months it has come from PaloAlto-assigned IP blocks. Previous to that they were using some Google CDN IPs. I have never seen any other Microsoft/etc. IPs or other sources associated with their scan. If there are not a lot of HTTP connections at the time I would start looking it NTP/DNS services are setup on your server. Using vulnerable services to generate oversize responses for DOS is quite common.

[aleksandar.astardzhiev]

Hi @PATRICKKASIE ,

I would agree with @Adrian_Jensen, but based on the access log it looks like the automatic scan is comming from Palo Alto product colled Cortex Xpanse - https://www.paloaltonetworks.com/cortex/cortex-xpanse  (Expanse is the old name of the product, which Palo Alto renamed when they aquired it - https://www.paloaltonetworks.com/company/press/2020/palo-alto-networks-completes-acquisition-of-expa... )

In a nutshell Xpanse can provide complete visibility of the public exposure for a company. Company provides its public IP range and Xpanse is running various automated scanns to find, identify and asses any public asset available in this public IP range.

Xpanse is not simple web crawler, so I would expect to be normal to experience heavier and more intrusive scan, than normal bots.

I would assume that someone who use this product has entered broader public range that inclused the IP assigned to your VPC and that way included your environment in the scan. I don't have personal experince with the product, but I could imagine everyone with active Xpanse subscription can do that (probably there is some at least some basic checks for IP range ownership, I hope).

As already suggested try to contact the email from the access log and request to remove your IP address from their scans.