AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
2.9K members online now
2.9K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

GoogleBot crawl issue beacuse of ? in URL

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hi, 

 

I am using woocommerce for my shopping website and have blocked URLs containing ? from being crawled via robots.txt. 

 

This however is affecting my merchant account as all of the product variants are getting disapproved due to their URL containing a ? e.g. https://website/product/abc/?attribute_pa_colour=gold. 

 

Is there a solution to this issue. 

 

Regards

 

1 Expert replyverified_user

GoogleBot crawl issue beacuse of ? in URL

Rising Star
# 2
Rising Star

@Pragya S

 

Hello.

Can you not stop blocking them in robots.txt?

GoogleBot crawl issue beacuse of ? in URL

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

Hi David, 

 

Yes we can but that would mean a lot of other URLs containing filers would be unblocked as well. 

 

User-agent: *
Disallow: /*?*

 

 

GoogleBot crawl issue beacuse of ? in URL

Rising Star
# 4
Rising Star

@Pragya S

 

I'm not familiar with a filer. What is that and why must it be blocked? Something SEO related? If so, you're going to have to figure out which is most important. The possible negative SEO effects, which are often over-hyped, or having a merchant account that works.  Assuming your concern is SEO related, instead of robots.txt, you might selectively choose which instances using ? are ignored in Webmaster Tools.  More details on that here: https://support.google.com/webmasters/answer/6080550

 

The only other thing I can think of is you need to find an Apache expert that's smart with RegEX, and have them create something for mod_rewrite for your product variants that removes the ? and makes it more friendly. For example, https://website/product/abc/attribute_pa_colour/gold

GoogleBot crawl issue beacuse of ? in URL

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭

Hi David, 

 

Thanks. Well Wocommerce and many other website use product filers which add a lot of variables & parameters to the URL. These URLS look like 

 

https://interiormantra.com.au/shop/?orderby=price&min_price=0&max_price=20 or 

https://interiormantra.com.au/product-category/wall-clocks/large-wall-clocks/?filter_colour=black 

 

Ofcourse we have blocked these URLS for the purpose of SEO however our concern is not SEO related. Moreover only some of these parameters can be selectively ignored in webmaster tools. The second url above is an example whose filer parameters cannot be ignored in google web masters as the parameter itself is 2 to 3 level deep and its depth does not always follow a pattern. 

 

The second solution is something we have already evaluated, however personally I am not a big fan of using RegEX to redirect URLS. 

 

Nevertheless thanks for your replies. Much appreciated. I guess I will talk to WP developers and see if they can help. 

 

Regards