topic Re: allow googlebot crawler only in General Topics

allow googlebot crawler only

bino150 — Wed, 05 Mar 2014 17:48:18 GMT

I found reference to this discussion but there were no details specifying how to allow googlebot crawls only.

I would like to only allow google and bing if that's possible. I can see options for google-analytics in the definition rules. Can this be done?

prb — Wed, 05 Mar 2014 21:57:27 GMT

I don't think if there is a specific application for googlebot crawlers. Will let someone answer this.

I am sure this is not an optimal solution, But did you consider using URL filtering, or Destination IP address/FQDN object?

bino150 — Fri, 11 Apr 2014 19:00:14 GMT

Sorry for the late reply prb. I decided for now to enable crawling. I thought perhaps there was more granular way to do it.

jdelio — Wed, 25 Feb 2015 16:49:22 GMT

This discussion is the closest that we can do at this time:

Past this, would be limiting it by IP only for application Web-Crawler.

The request for a reverse lookup or a way to control the Spider/web crawlers would be a feature request that you need to talk with your SE about.

I hope this helps.

-JoeD