AdWords
2.8K members online now
2.8K members online now
Understand Google's advertising policies, including ad approval status and account suspension
Guide Me
star_border
Reply

Product pages cannot be crawled because of robots.txt restriction

[ Edited ]
Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

I've spent days trying to figure out what the issue is. I've even contacted my web provider, BigCommerce and they can't figure out what's wrong, either. I've even added the snippet of code Google recommends to allow crawling and still nothing. Any ideas?

1 Expert replyverified_user

Product pages cannot be crawled because of robots.txt restriction

Top Contributor
# 2
Top Contributor

Hi @mark o,

 

Can you copy your robots.txt info here for us to see? Have you looked in Google Search Console (Webmasters) and viewed the robots.txt and the blocked resources?

 

Kind Regards,

 

James

_________________________________________________________
Google AdWords Top Contributor | Google Partner | GYBO | Local Guide | My Profile


 


 


 

Product pages cannot be crawled because of robots.txt restriction

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

Here it is.

 

User-agent: AdsBot-Google
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /__socialshop/

User-agent: *
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /__socialshop/

Product pages cannot be crawled because of robots.txt restriction

Top Contributor
# 4
Top Contributor

Hi @mark o,

 

You would need to change the first half to:

 

User-agent: AdsBot-Google
Disallow:

 

Blocking the Google Adsbot brings different issues and includes what you are experiencing. There is no harm in allowing the Google Adsbot to crawl the areas you are attempting to block them from with your robots.txt.

 

Kind Regards,

 

James 

_________________________________________________________
Google AdWords Top Contributor | Google Partner | GYBO | Local Guide | My Profile


 


 


 

Product pages cannot be crawled because of robots.txt restriction

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭

Thank you, I'll make the change.

Product pages cannot be crawled because of robots.txt restriction

Top Contributor
# 6
Top Contributor

Hi @mark o,

 

You are very welcome. Please let us know how things go.

 

Kind Regards,

 

James

_________________________________________________________
Google AdWords Top Contributor | Google Partner | GYBO | Local Guide | My Profile