AdWords
1.8K members online now
1.8K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

Google Products Disapproved or Invalid Robots.txt & images ok??

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Please can you help me out, ive been searching many of the forums.  My 219 items are inserted into google products and 219 have shown as disapproved or invalid. When i check the Data Qaulity it says Critical error images cannot be crawled because of robots.txt restriction(100% of all items are affected 18 out of 18 items), which is not a very logical message anyway because I have 219 items that are dissaproved or invalid and now there are messages about 18??? 

 

Anyway, after checking data quality in the dashboard products overview, it looks like they were approved for about 12 hours and now they are all disapproved again.

 

I have resubmitted the feed and still no good.  The images are all the correct size (i.e. not more than 4gb and over 800 pixels)

 

This is my robots.txt file 

 

# $Id: robots.txt,v magento-specific 2010/28/01 18:24:19 goba Exp $
#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these “robots” where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used: http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html
# Sitemap: http://www.mywebsite.com/sitemap.xml

# Crawlers Setup
User-agent: *
Disallow:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
# Crawl-delay: 30
# Allowable Index
Allow: /*?p=
Allow: /index.php/blog/
Allow: /catalog/seo_sitemap/category/
Allow: /catalogsearch/result/
# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
# Paths (clean URLs)
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=
Disallow: /*?limit=all

# Website Sitemap
Sitemap: http://www.thelinenbin.com/sitemap.xml

# Uncomment if you do not wish for Google to index your images
#User-agent: Googlebot-Image
#Disallow:

 

Pleas can someone help?

1 Expert replyverified_user

Re: Google Products Disapproved or Invalid Robots.txt & images ok?

[ Edited ]
Top Contributor
# 2
Top Contributor

the data-quality messages are samples (examples);
the out-of message reflects the number of items that
were sampled by google and are also being flagged --
not the number of items submitted in the data (feed);
the full extent of the issue may be much more expansive.


flagged images can be for many image related issues --
including the actual size of the product image vs the size
of the entire image (including any borders), any text seen
within the image, more than one product displayed, etc.


the robots message may also appear for many issues related
to the webserver serving the image -- for example, any error
response codes returned to google at the time of the crawl;
for example, the website (server) not being able to keep pace
with all googlebot and googlebot-image crawl requests --
especially if the images are too large with respect to the
given server and bandwidth resources, etc.

any item that does not have a proper image for google
to display (regarless of the reason) can indeed trigger a
disapproval -- however, a single disapproval can be for
many multiple issues (many having nothing to do with

any image issue or with the feed itself).

unfortunately, forum members cannot look into feeds or accounts --
if you would like to post more exact details here in the public forum

we may be able to offer more exact suggestions or possibilities.

 

"disapproved again."

google may sometimes take 24-72 hours or so before all items

in a submitted feed are inspected and have a quiescent status --

and the website has been crawled for any quality related issues.

 

an email from google should have been sent

regarding the disapproval issue -- otherwise,
google may be contacted directly here --
https://support.google.com/merchants/?hl=en#topic=3404818&contact=1

see also
https://www.en.adwords-community.com/t5/Google-Shopping-PLAs-Merchant/Merchant-Center-account-suspen...
https://support.google.com/merchants/answer/188484
https://support.google.com/merchants/answer/160491
https://support.google.com/merchants/?hl=en#topic=3404779

as to the robots.txt specifically, regardless of what might seem well
or might have worked in the past (or until the issue is resolved) try
adding the following nine records (lines) to the very end of the site's
controlling robots.txt
#

User-agent: Googlebot-image
Disallow:

User-agent: Googlebot
Disallow:

#

Re: Google Products Disapproved or Invalid Robots.txt & images ok?

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

Eureka:

 

After much hair pulling I have just discovered that all the disapproved images in my google feed are due to the fact that I copied and pasted my site image URLs into the feed using HTTPS instead of HTTP.

 

I know this mistake is elementary to the point of being stupid, but it is so obvious a mistake that I simply did not see it . Added to which all the images in my feed with the incorrect HTTPS were all valid images but not crawable due to the incorrect prefix.

 

I spent many fun filled hours trying to understand why my Robts.txt file was deficient when it technically was not. It was the image URL that was not correct.

 

It is a great pity that no Google documuntation nor advice from Adwords support ever mentioned this as a possible error.

 

Maybe this "discovery" will help others.