AdWords
2.7K members online now
2.7K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

GoogleBot crawl issue

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

hello,

 

Googlebots are absolutely hammering our site and I dont know how to stop it. I have off-grid-europe.com/robots.txt, what is wrong with this? How do I stop this?

 

How do I get google just to normally craw the site,  and not try and find every link possible without blocking my merchant feed from being crawled?

 

This is the advice given my google to prevent turning the bots away.

User-agent: Googlebot-Image
Disallow:
User-agent: Googlebot
Disallow:

I also reduced the crawlrate to 30 seconds on webmaster tools which helped a bit but now the problem is back. And I have a sitemap to index in the robots.txt  which really is all I want robots to index period.


This problem didnt always happen on the site, It began in August. Any idea why this is?

 

Also, I have been going around circles here

https://productforums.google.com/forum/?utm_medium=email&utm_source=footer#!msg/webmasters/wD4ChY4ZV...

 

Regarding this problem, no solution

1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by pankaj1782 (Top Contributor)
September 2015

Re: GoogleBot crawl issue

Participant ✭ ✭ ☆
# 2
Participant ✭ ✭ ☆
Hi Christiane,

Unfortunately our experience is in AdWords, primarily. The Webmaster forum is probably your best bet for getting this figured out.

good luck!

Best,
Amy

View solution in original post

Marked as Best Answer.
Solution
Accepted by pankaj1782 (Top Contributor)
September 2015

Re: GoogleBot crawl issue

Participant ✭ ✭ ☆
# 2
Participant ✭ ✭ ☆
Hi Christiane,

Unfortunately our experience is in AdWords, primarily. The Webmaster forum is probably your best bet for getting this figured out.

good luck!

Best,
Amy