AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
4.5K members online now
4.5K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

[Google Merchant]: page not crawlable due to robots.txt

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hi, I have a Prestashop based ecommerce and I uploaded some products (20) in google merchant to test it before uploading all the others.

 

6 of those products are disapproved: 3 of them because "Desktop page not crawlable due to robots.txt" and 3 of them because "Mobile page not crawlable due to robots.txt". The "?" sign next to the error message read: "Update your robots.txt file to allow user-agents "Googlebot" and "Googlebot-Image" to crawl your site"

 

So I changed my robots.txt file to allow both googlebot and googlebot-image, following the istruction i found on this forum, adding:
---------------------------------------------------------
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:

---------------------------------------------------------

 

Then i tested it with Google search console, trying the links of the disapproved products, ad everything seems to be ok, it says they are allowed.
But they still are disapproved. I wonder why 6 items are disapproved and 14 are active since the page is basically the same, and why it says that the page is not crawlable when they are allowed.

Any ideas how to fix that? 

1 Expert replyverified_user

[Google Merchant]: page not crawlable due to robots.txt

Explorer ✭ ✭ ✭
# 2
Explorer ✭ ✭ ✭

Hi, 

 

Same issue (solved) here : https://www.en.advertisercommunity.com/t5/Google-Ad-Grants/Product-pages-cannot-be-crawled-because-o...

 

Try it and let us know if it works :-)

[Google Merchant]: page not crawlable due to robots.txt

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

Did you manage to fix it ? We have the exact same issue at the moment which popped up randomly and struggling to find a fix

[Google Merchant]: page not crawlable due to robots.txt

Top Contributor
# 4
Top Contributor

Hi The Friday F,

 

Did you visit the link supplied by Bruno? That would be your best course of action since it does provide very detailed information that will likely resolve your issue. 

 

Kind Regards,

 

James

____________________________________________________________
Google My Business & Google Ads Top Contributor
Google Partner | GYBO Agency | Local Guide | My Profile