topic Re: Spurious hits from the Expanse webcrawler... in General Topics

Spurious hits from the Expanse webcrawler...

OtherWorldsTV — Fri, 12 Nov 2021 17:49:07 GMT

Much the same as the issue in this post: https://live.paloaltonetworks.com/t5/general-topics/incoming-traffic-from-palo-alto-ip-address/td-p/196099 only with a different set of IP addresses (34.77.162.0 - 34.96.130.0).

Telling me that "we crawl on a regular basis" is decidedly NOT an answer!

One, I am not a client of Palo Alto or Expanse, Inc, so your crawling is of no benefit to myself.
Two, your "regular crawling" is disrupting my network majorly. Is that really your end goal?

What EXACTLY are you looking for on my PRIVATE network?

Partial log:

MISS|403|1636727367297|2668|604509|34.86.35.0|-|http://cdnotherworlds.b-cdn.net/|IL|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|efd0e52587a035b40b4f0568448b0507|US
MISS|403|1636718015879|2668|604509|34.77.162.0|-|http://cdnotherworlds.b-cdn.net/|TX|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|38ec458cac0e000c52a54a3fdc02b407|US
MISS|403|1636711019886|2668|604509|34.77.162.0|-|https://cdn.otherworlds.tv/|MI|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|00963a032da5da4b32c7382d1e6bc10e|US
MISS|403|1636710780963|2668|604509|34.86.35.0|-|https://cdn.otherworlds.tv/|DE|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|87b25183aaa7f90ea9074e09ef68e9e7|US
MISS|403|1636709293552|2668|604509|34.96.130.0|-|http://cdnotherworlds.b-cdn.net/|DE|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|3e6c123ef8b8cbd65bd938d4decad824|US
MISS|403|1636697076710|2668|604509|34.96.130.0|-|https://cdn.otherworlds.tv/|LA|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|199bcf329b04eed7cd8112c05f1d75ee|US
MISS|403|1636693708796|2668|604509|34.77.162.0|-|http://cdn.otherworlds.tv/|LA|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|95c210d97d6db0993d277bffecace1d2|US
MISS|403|1636682456854|2668|604509|34.77.162.0|-|https://cdnotherworlds.b-cdn.net/|DE|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|aaf73b49b6c569d067c921a8a88a730c|US
MISS|403|1636681389575|2668|604509|34.86.35.0|-|https://cdnotherworlds.b-cdn.net/|MI|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|e4d4cbaeb257d24a6dcfc9b4ca7787cd|US
MISS|403|1636673213470|2668|604509|34.77.162.0|-|http://cdn.otherworlds.tv/|DE|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|800a4e888d49bd473492fd4dcd234ff7|US
MISS|403|1636672374657|2668|604509|34.77.162.0|-|https://cdnotherworlds.b-cdn.net/|IL|Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com|ed1cc263aadda33463bcecc10ee7b9c3|US

Re: Spurious hits from the Expanse webcrawler...

aleksandar.astardzhiev — Fri, 12 Nov 2021 18:35:01 GMT

Hi @OtherWorldsTV,

- It may be not beneficial to you, but it is definately beneficial to others. If you read the all posts from the link you found you will understand that Palo Alto - like any other vendor with URL filtering product - is keeping a "database" of websites/domains/urls and try to categorise them. That way Palo Alto products can be used as URL filtering solution and control access to websites, based on their content/category. One way to keep such database is to use automated crawlers. This crawlers will try to crawl any domain that is currently not categorized - "unknown" and give it proper category. So you see why it is beneficial for other to crawl your domain - so it can be categorized and Palo Alto product users to safely access your domain.

- I am not sure what you mean by "private network". If your domain is publically resolvable and there is no access control (any public IP can access it), that it is obviosly not a private network. If it ment to be private, than you are responsible to put the proper access control and restrict the access. Web crawling is very common think these days, how do you think web searches like Google, Bing etc are indexing the whole internet? Do you think Palo Alto Networks crawlers are the only one crawling your domain?

- What are your concerns? How does those crawlers disrupt your network?

- Have you tried to contact the email from the user-agent and explain your problem?

Re: Spurious hits from the Expanse webcrawler...

OtherWorldsTV — Fri, 12 Nov 2021 19:13:37 GMT

I'm sorry, but rather than making excuses for their bad behavior, you should be asking WHY their crawler is hitting my server 20,000 times since Midnight causing a Denial of Service.

The server in question is not a web server, but a file server for a Roku channel, which has zero use for any Palo Alto product user.

Other crawlers (Google, Bing, etc) all follow the directives set by our robots.txt and do NOT try to index our servers. Why should Palo Alto be any different?

As for contacting them by email, I've tried that. And got an immediate bounce message saying my email was "undeliverable."

Honestly, a single scan and move on would be fine. But continually hitting my server this excessively shows malicious intent on the part of Palo Alto Networks.

Re: Spurious hits from the Expanse webcrawler...

OtherWorldsTV — Fri, 12 Nov 2021 19:28:56 GMT

Their phone support is equally impressive. I've only been hung up on 3 times, and spent a little over 2 hours on hold so far.

Re: Spurious hits from the Expanse webcrawler...

OtakarKlier — Mon, 15 Nov 2021 16:02:08 GMT

Hello,

Yeah sounds a bit off to me as well. Definitely try to get someone on the phone, since you are not a customer:

Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com

Also an email? In the meantime, set your firewall to drop the traffic?

Regards,

Re: Spurious hits from the Expanse webcrawler...

OtherWorldsTV — Mon, 15 Nov 2021 19:31:38 GMT

By default we already drop any traffic that doesn't meet specific patterns. But it's annoying watching the intrusion logs fill up with Expanse over and over.

As I said, their phone support lives up to everything else I've seen from them so far. 2 hour hold times, repeated hang ups, and one particular call of being taken off hold to be asked "Are you sure you aren't a customer" FIVE times before being hung up on.

Honestly, I think they should be investigated for criminal intent. Their "research" is doing nothing to benefit anyone.