AdWords
2.1K members online now
2.1K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

Images cannot be crawled because of robots.txt restriction

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

hi,

 

I uploaded a product feed to google merchant center, but got the error:Images cannot be crawled because of robots.txt restriction

 

robots.txt in root directory is:

User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
User-agent: AdsBot-Google
Disallow:

 

Why I still got the crawl error? Thank you.

1 Expert replyverified_user

Images cannot be crawled because of robots.txt restriction

Rising Star
# 2
Rising Star

Make sure that you have given Google enough time after you made changes. It takes up to 30 days for Google's diagnostic tab to clear any old issues. As Google does not crawl every page every day, but spread out in 30 days.

 

Hope it helps

Twitter | Linkedin | Community Profile | Shopping Feed Tips From FeedArmy
Did you find any helpful responses or answers to your query? If yes, please click on ‘Accept As Solution’

Images cannot be crawled because of robots.txt restriction

Explorer ✭ ✭ ☆
# 3
Explorer ✭ ✭ ☆

Hi Strovita,

Simply remove

 

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

 

from your robots.txt file.

Images cannot be crawled because of robots.txt restriction

Visitor ✭ ✭ ✭
# 4
Visitor ✭ ✭ ✭

Do you use a CDN to serve images? If that is the case make sure your CDN is not blocking Googlebot and it has the copy of your robots.txt.

 

Go check in Search Console (Webmaster Tools) - Fetch as Google, if your images are being crawled correctly.

 

We had a problem on one site where the client had two CDNs (MaxCDN for serving images and CloudFlare for some other features). What happened?! Googlebot requested an image from MaxCDN which requested Cloudflare for permission. And CloudFlare blocked the request because it was not a valid GoogleBot IP address. It was MaxCDNs IP address.

 

Hope this will be helpful.

Images cannot be crawled because of robots.txt restriction

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭

On Feb.16, there are two errors there. One is "Images cannot be crawled because of robots.txt restriction", the other is no GTIN for products.

I have added GTIN, and upload the feed again.

Now GTIN error is gone, so I think new feed file is already checked. But why "Images cannot be crawled because of robots.txt restriction" is still there? The images are not crawled again. The last time crawled is still Feb. 14

Images cannot be crawled because of robots.txt restriction

Visitor ✭ ✭ ✭
# 6
Visitor ✭ ✭ ✭

We have a CDN. But I check in Search Console (Webmaster Tools) - Fetch as Google, the image link could be fetched

Images cannot be crawled because of robots.txt restriction

Rising Star
# 7
Rising Star

When you make a change, you need to give google a couple of weeks to determine you have fixed the issue, if you are using the diagnostics tab as your reference. You can validate if the issue is resolved if the number of issues against the warning is dropping down.

 

If you see the error in the diagnostics tab it takes up to 30 days to have all the errors removed even when you have fixed the issue. Google does not crawl every page every day.

 

If the issue persist, and you have waited at least 2 weeks than revisit the help docs to resolve the issue.

https://support.google.com/merchants/answer/6098154?hl=en

 

Hope it helps

 

 

Twitter | Linkedin | Community Profile | Shopping Feed Tips From FeedArmy
Did you find any helpful responses or answers to your query? If yes, please click on ‘Accept As Solution’