AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
1.9K members online now
1.9K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

robots.txt and google crawl rate

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

hello,

 

Googlebots are absolutely hammering our site and I dont know how to stop it. I have off-grid-europe.com/robots.txt, what is wrong with this? How do I stop this?

 

How do I get google just to index the sitemap and not try and find every link possible without blocking my merchant feed from being crawled?

 

As you can see I allowed google bots to index my site

User-agent: Googlebot-Image
Disallow:
User-agent: Googlebot
Disallow:

I also reduced the crawlrate to 30 seconds on webmaster tools which helped a bit but now the problem is back. And I have a sitemap to index in the robots.txt  which really is all I want robots to index period.


This problem didnt always happen on the site, It began in August. Any idea why this is?

 

Kind regards,

Mark

 

1 Expert replyverified_user

Re: robots.txt and google crawl rate

Top Contributor
# 2
Top Contributor
Hi Christiane,
I think you'd get a better answer over on the Webmaster Central forum. This is the AdWords forum so the guys over here will be able to answer your question quicker and more thoroughly https://productforums.google.com/forum/#!forum/webmasters

Hope that helps.