allow googlebot crawler only

bino150 · ‎03-05-2014

I found reference to this discussion but there were no details specifying how to allow googlebot crawls only.

I would like to only allow google and bing if that's possible. I can see options for google-analytics in the definition rules. Can this be done?

prb · ‎03-05-2014

I don't think if there is a specific application for googlebot crawlers. Will let someone answer this.

I am sure this is not an optimal solution, But did you consider using URL filtering, or Destination IP address/FQDN object?

bino150 · ‎04-11-2014

Sorry for the late reply prb. I decided for now to enable crawling. I thought perhaps there was more granular way to do it.

jdelio · ‎02-25-2015

This discussion is the closest that we can do at this time:

Past this, would be limiting it by IP only for application Web-Crawler.

The request for a reverse lookup or a way to control the Spider/web crawlers would be a feature request that you need to talk with your SE about.

I hope this helps.

-JoeD

LIVEcommunity team member
Stay Secure,
Joe
Don't forget to Like items if a post is helpful to you!