allow googlebot crawler only

Reply
Highlighted
Not applicable

allow googlebot crawler only

I found reference to this discussion but there were no details specifying how to allow googlebot crawls only.

How to Allow Googlebot Through the Firewall

I would like to only allow google and bing if that's possible. I can see options for google-analytics in the definition rules. Can this be done?

Highlighted
L3 Networker

I don't think if there is a specific application for googlebot crawlers. Will let someone answer this.

I am sure this is not an optimal solution, But did you consider using URL filtering, or Destination IP address/FQDN object?

Not applicable

Sorry for the late reply prb. I decided for now to enable crawling. I thought perhaps there was more granular way to do it.

Highlighted
Community Team Member

This discussion is the closest that we can do at this time:

Re: web crawling for google only

Past this, would be limiting it by IP only for application Web-Crawler.

The request for a reverse lookup or a way to control the Spider/web crawlers would be a feature request that you need to talk with your SE about.

I hope this helps.

-JoeD

Stay Secure,
Joe
End of line
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!