web crawling for google only

Reply
Highlighted
Not applicable

web crawling for google only

I know this topic has been discussed before but there is never a clear answer. It seems it is not possible to allow only specific web crawlers such as google. If that's the case, I assume most of you have web-crawling enabled for your site only? Google is still getting blocked from crawling our site. I was hesitant to enable web-crawling but it sounds like that's the only way its going to work.

L7 Applicator

There is no easy way to have the googlebot identified in a rule so that it would be the only one allowed to crawl.

Google does provide a way to verify the bot via reverse dns lookup of the bot ip address.  Here you check the source of the crawler and lookup in DNS to a specific google subdomain as outlined in this document.  The issue is there is no way to have this kind of check in a PA rule at this point.

Verifying Googlebot - Webmaster Tools Help

Google bot also uses a specified user agent string.  So you could create a custom vulnerability signature to look for hte string.  There are two issues with this method.  One it is a good way to block but not a way to permit only the hits.  And two there are people faking these user agents since they know they are trusted and would pass the test even without being the real deal.

Google crawlers - Webmaster Tools Help

Steve Puluka BSEET - IP Architect - DQE Communications (Metro Ethernet/ISP)
ACE PanOS 6; ACE PanOS 7; ASE 3.0; PSE 7.0 Foundations & Associate in Platform; Cyber Security; Data Center
Highlighted
Not applicable

Thank you.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!