About Rags

Anything for  PCNSA?   Thank you,   -Rags ... View more

I could not get the OpenVAS signature to work by just switching Nikto for OpenVAS. I had to do a more basic string <pattern>OpenVAS</pattern> . I actually did <pattern>OpenVAS 8</pattern> at first to see if that would work, it did. (OpenVAS 8.0.9 was the user agent.    Make sure you edit the entry name if you use my .xml file. You might already have that number in use. Also created one for Baiduspider since my IDS picked it up in a scan and I saw the user agent string for it.    There's a few useful links that will show user agent strings for popular scanners/crawlers.    https://developers.whatismybrowser.com/useragents/explore/software_type_specific/crawler/9 http://www.useragentstring.com/pages/useragentstring.php?typ=Crawler   Hope that helps, thanks -Rags ... View more

As BPry said above, the 2 bottom IPs are probably going through something hosted by akamai. I see them listed as zscaler, they might be a cloud platform hosted on akamai.    https://technet.microsoft.com/en-us/library/bb693717.aspx Same list as above ... View more

Due to this issue, they have now changed the matric (matric?? metric?) crawlers so that it should only be going out on 70.42.131.170. This would be the IP they can look for. Let us know if there is anything else or if we may close the case. So, based on the above, PAN is only scanning from 70.42.131.170, so you are seeing traffic that can be expected. At least now we have an official response from Palo Alto on the issue and they have confirmed that it is indeed them doing the scanning from 70.42.131.170.   At this point I'm still not sure of which action to take. I kind of don't want to block the IP if it is categorizing our newly hosted URLs. At least we know what it is now.   Thanks, -Rags ... View more

@MatthiasBehn Has the traffic subsided for your customer?   Thanks, -Rags ... View more

We received a decent response from PAN, butI don't believe these issues are related. This traffic started months ago and is still active, while the response indicates it has been resolved.    On Tuesday, February 6th, 2018 we became aware of IPS events being triggered in some customer environments with source IP addresses attributable to Palo Alto Networks. We determined these events were related to benign scanning by the Palo Alto Networks URL Filtering service. Please note, this did not pose any risk to your organization's security.  PAN-DB relies on a set of systems designed to automatically identify, crawl, and categorize content on the web. For these IPS events, we observed HTTP activity in some customer environments generated lookups to URL Filtering systems for 'unknown' URL's. Based on this information, our crawlers visited systems with IP addresses associated with those URL lookups. These routine attempts to visit and categorize URLs unknown to the cloud triggered some IPS systems to inadvertently identify the traffic as command-and-control activity, based on a unique hardcoded string in the request. On Thursday, Feb 8 at 1:30PM PST we implemented a fix to prevent lookups from our PAN-DB web crawlers from triggering IPS signatures in the future. We regret any inconvenience caused by these false-positives and thank you for your patience as we worked to resolve the issue. Please be assured that this did not pose any risk to your organization's security and these events were not related to any attacks from Palo Alto Networks. ... View more

Thanks @kalakai for the links. Good stuff.   What is the advantage over using a sinkhole over just using a DBL and setting the rule to block completely?   Thanks, -Rags ... View more

Thanks for the information, will do.   > give us a list of your URLs/domains/IPs, so we can exclude them from analysis/scan.   The reason I don't want to do this is because we often have developers creating new domain names. If your crawlers do not crawl our network for these newly hosted domains, then they will be listed as unknown and the blocked since we have that category set to block.    Take care, -Rags ... View more

Thanks for the additional information @khuynh   >I have in my region several customers who have the same concerns as you   At this point what solutions do you provide for them? If you have previously asked them to open a TAC was there any resolution found that you can share? I really just want to know if this is a single 1-off IP Address, or if this web crawling traffic will be coming from an entire PAN IP Address subnet. We can then create our own settings that would be best to mitigate this traffic.   Thanks, -Rags ... View more

@khuynh That's a really laxed response for a security related incident that you're responsible for. Why would I need to make a TAC when you're the person who is apparently involved in the scanning/web crawling activity? PAN should have this published to customers so we do not have to use resouces to researching a potentional security issue.    Thanks I will make a TAC, and I have changed my mind and removed you from my friends list.    Regards, ... View more

Would it be reasonable to ask PAN to release the IP Range of IPs that they use for web crawling/scanning? We could create our own security policy so these events won't log and note that it is our security vendor crawling us.    Thanks, Rags ... View more

image. ... View more

@khuynh   This IP Address has been spamming my network, 4.5 million events since early october~~. This is setting off IDS alerts and taking away time from our team to research. This seems really excessive for a web crawler does it not? It's constant 35,000 - 39,000 events per day.   Can you stop, or do we just blacklist you?  ... View more