AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
2.7K members online now
2.7K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

page not crawlable due to robots.txt

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hello there,

Yesterday I've got 57% of my products disapproved in google merchant. The error is:

Mobile page not crawlable due to robots.txt
Desktop page not crawlable due to robots.txt

Could you help me with those issues?

My robots.txt below:

 

# we use Shopify as our ecommerce platform

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /5309005871/checkouts
Disallow: /5309005871/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*design_theme_id*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /discount/*
Disallow: /gift_cards/*
Disallow: /apple-app-site-association
Sitemap: https://mywebsite.com/sitemap.xml

# Google adsbot ignores robots.txt unless specifically named!
User-agent: adsbot-google
Disallow: /checkout
Disallow: /carts
Disallow: /orders
Disallow: /5309005871/checkouts
Disallow: /5309005871/orders
Disallow: /discount/*
Disallow: /gift_cards/*
Disallow: /*design_theme_id*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*

User-agent: Nutch
Disallow: /

User-agent: MJ12bot
Crawl-Delay: 10

User-agent: Pinterest
Crawl-delay: 1

 

1 Expert replyverified_user

page not crawlable due to robots.txt

[ Edited ]
Visitor ✭ ✭ ✭
# 2
Visitor ✭ ✭ ✭

I have the same problem today and my shopping ads are disapproved, i haven't even adjusted my robots.txt, is there a fix ?

page not crawlable due to robots.txt

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

I'm having the same problem with one of my clients on Shopify. Of course with Shopify you can't modify the robots.txt and there are no inline meta robots directives that could be triggering the issue. Has happened roughly every 7-14 days in Google Merchant Center over the past 30 days, with the worst yesterday blocking almost 40% of active products that were previously approved.

page not crawlable due to robots.txt

[ Edited ]
Top Contributor
# 4
Top Contributor

the best likely course is to contact

shopify support directly and to also

possibly post in the shopify forums --
https://www.shopify.com/contact

https://help.shopify.com/questions

https://ecommerce.shopify.com/forums