AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
1.7K members online now
1.7K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

Google Merchant crawling from CHINA and getting blocked

FM
Visitor ✭ ✭ ✭
# 1
FM
Visitor ✭ ✭ ✭

I noticed:

"Product pages cannot be accessed from a mobile device"

on our "Data quality" page due to a single product that supposedly got a 403 on a product url...

 

seemed very unlikely...

checked my logs....

203.208.60.170 - - [08/Sep/2014:...] "GET PRODUCT_URL  HTTP/1.1" 403 290 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

[Mon Sep 08 ... 2014] [error] [client 203.208.60.170] client denied by server configuration: PRODUCT_URL

 

and 203.208.60.170 is from CHINA

 

we currently block ALL of china due to the massive number of crackers originating from the country

I have read on other web sites that this was a bug that was fixed back in 2011, have policies changed?  Is this a bug that needs reporting?

 

For now I will unblock only Google.

1 Expert replyverified_user

Re: Google Merchant crawling from CHINA and getting blocked

Community Manager
# 2
Community Manager
Hi FM,

I am going to look into this for you. I'll be back in touch. If this is account specific, I'll send you an email.

Kathleen

Re: Google Merchant crawling from CHINA and getting blocked

Participant ✭ ✭ ☆
# 3
Participant ✭ ✭ ☆

Hi,  If your blocking all Chinese based IP's then your also blocking any google crawlers that maybe housed there. 

 

The ip appears to be from china: http://mxtoolbox.com/SuperTool.aspx?action=ptr%3a203.208.60.170&run=toolpage and also identifies  as being Google owned. 

 

I'm not sure how you've configured the blocking it appears through apache given the log entries.  

You might want to switch to only blocking known problematic ip's instead of a whole country given the problems your having. 

 

You can also verify if the ip is googlebot to a degree here : https://support.google.com/webmasters/answer/80553

 

Theres a general article about making sure GoogleBot isn't blocked here: https://support.google.com/webmasters/answer/2387297?hl=en

 

Tom Wilson
Freelance Adwords & Google Shopping Consultant

If you find this or another response helpful within the forum "please accept as solution" to further assist and direct others.