6.6K members online now
6.6K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply
Highlighted

Shopping Ad dissapproved - Timeout reading robots.txt

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

3 of my 20 Google Merchant (Shopping) product ads were disapproved on Jan 11th for two reasons: a) Timeout reading robots.txt OR b) robots.txt on the server is unreachable.  Per conversation with AdWords customer representative on Jan 12th, I added the following lines to my robots.txt (although nothing previously there suggested that the Google bots should have been blocked).

 

User-agent: Googlebot
Disallow:

User-agent: AdsBot-Google
Disallow:

User-agent: Googlebot-Image
Disallow:

 

When I use the Search Console to 'robots.txt Tester', everything seems fine.  My website is on a managed server (by professional hosting people, who presumably know what they are doing).  They are equally stumped as to why Google's bot claims it can't reach three of my pages.  I did change my data-feed today to point to a different product page within the website, just to see if that helps?  I received multiple orders during the time-stamp periods that Google said it couldn't access the robot.txt files, suggesting that the site wasn't actually down (and confirmed by my hosting provider).

 

Not sure if I should "wait-it-out" or failover to my backup servers, or something else since Google's bot seems to be the only tool that can't access my site.

1 Expert replyverified_user
Marked as Best Answer.
Solution
Accepted by topic author brian a
2d

Shopping Ad dissapproved - Timeout reading robots.txt

Rising Star
# 2
Rising Star

This can happen due to several reasons

A DNS has failed between Google's Bot and the Server IP's there can be up to 15 or 20 hops depending on the server location and connections

The server was overloaded and did not respond in a timely manner

The server was temporarily down

 

I recommend looking into your server logs and check for any issues.

 

To resubmit the products, change the image file or and a random query string at the end of the url.

 

This should trigger a new crawl.

 

Otherwise wait until Google crawls the product again which can take up to 30 days.

 

Hope it helps.

Shopping Feed Tips From FeedArmy
Find me on: Twitter | YouTube | Facebook
Did you find any helpful responses or answers to your query? If yes, please mark it as the Best Answer

Shopping Ad dissapproved - Timeout reading robots.txt

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

I appreciate the insight.

 

In addition to updating the robots.txt (previous post), I did change the product-landing-page for all the ads, and fetched the new feed in hopes that triggers a review/crawl.

 

I'm just not looking forward to informing our hourly employees when they return from the long-weekend, that their hours may be reduced by 30%, until Google's fake-person (a bot) can properly communicate with our website's electrons.

 

Employee: "But isn't there a human-being you can talk to over the phone to help trouble-shoot?  I need the work-hours".

Me: "I'm sorry, but we need to wait until Google's computer automatically re-crawls the site... which could take up to 30-days... or longer if the actions we took didn't remediate the issue."

Employee: "But surely, there is some way research the issue."

Me: "There is.  We have confirmed that their bots are able to successfully access the page through some tools called 'Search Console', 'Page Speed Insights', and 'Mobile Friendly Test'.  We'll all just have to wait together."

Shopping Ad dissapproved - Timeout reading robots.txt

Visitor ✭ ✭ ✭
# 4
Visitor ✭ ✭ ✭

Update: 2 of the 3 originally disapproved product ads an now active again.  It took about 2.5 days for the re-crawl to activate.