topic Re: Excessively long useragent in General Topics

Excessively long useragent

chaeron — Fri, 01 Sep 2023 09:56:24 GMT

I don't think the following bot useragent is acceptable:

Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com

It is excessively long, and reads more like an advert. A legit organisation should not be doing this IMO

Site admins should need to opt-in, as is good practice for accepting cookies for example, not opt-out.

Does the Palo bot abide by robots.txt or similar mechanism?

Re: Excessively long useragent

kiwi — Wed, 06 Sep 2023 10:54:16 GMT

Hi @chaeron ,

This was discussed before here:

https://live.paloaltonetworks.com/t5/general-topics/high-loads-by-scanner/td-p/503445

As suggested, in order to be removed contact the email from the access log and request to remove your IP address from the scans.

Kind regards,

-Kim.

Re: Excessively long useragent

chaeron — Wed, 06 Sep 2023 12:25:55 GMT

Thank you for your response.
Although I understand site owners can request that the Palo bot does not scan their ip address(es), this sets a precedent, site owners should not be expected to "opt out" via email from bots sent from legitimate companies - that would be a full-time job 🙂
At a minimum, the robots.txt file should be respected.
...non-legit organisations would ignore the request, and possibly spam harder ( how does a site owner know Palo is legit, again, researching this would be part of the full-time job )
I think my main point is that the site logs get populated with overly-long "useragents" that read like advertisements, something a malicious actor might do, but not a legitimate organisation.
Looking at the multitude of useragents, malicious and legit, Palo bot stands out as not really being a useragent at all.
Although not a scientific test, a sample of 10,000 useragents gives me an average string length of 78, Palo bot is 269 characters in length.
imo legitimate companies should "play nicely" and follow convention.