Google Products Disapproved or Invalid Robots.txt & images ok??
Please can you help me out, ive been searching many of the forums. My 219 items are inserted into google products and 219 have shown as disapproved or invalid. When i check the Data Qaulity it says Critical error images cannot be crawled because of robots.txt restriction(100% of all items are affected 18 out of 18 items), which is not a very logical message anyway because I have 219 items that are dissaproved or invalid and now there are messages about 18???
Anyway, after checking data quality in the dashboard products overview, it looks like they were approved for about 12 hours and now they are all disapproved again.
I have resubmitted the feed and still no good. The images are all the correct size (i.e. not more than 4gb and over 800 pixels)
This is my robots.txt file
# $Id: robots.txt,v magento-specific 2010/28/01 18:24:19 goba Exp $
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these â€œrobotsâ€ where not to go on your site,
# you save bandwidth and server resources.
# This file will be ignored unless it is at the root of your host:
# Used: http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
# For more information about the robots.txt standard, see:
# For syntax checking, see:
# Sitemap: http://www.mywebsite.com/sitemap.xml
# Crawlers Setup
# Crawl-delay: 30
# Allowable Index
# Paths (clean URLs)
# Paths (no clean URLs)
# Website Sitemap
# Uncomment if you do not wish for Google to index your images
Pleas can someone help?
Re: Google Products Disapproved or Invalid Robots.txt & images ok?[ Edited ]
March 2014 - last edited March 2014
the data-quality messages are samples (examples);
the out-of message reflects the number of items that
were sampled by google and are also being flagged --
not the number of items submitted in the data (feed);
the full extent of the issue may be much more expansive.
flagged images can be for many image related issues --
including the actual size of the product image vs the size
of the entire image (including any borders), any text seen
within the image, more than one product displayed, etc.
the robots message may also appear for many issues related
to the webserver serving the image -- for example, any error
response codes returned to google at the time of the crawl;
for example, the website (server) not being able to keep pace
with all googlebot and googlebot-image crawl requests --
especially if the images are too large with respect to the
given server and bandwidth resources, etc.
any item that does not have a proper image for google
to display (regarless of the reason) can indeed trigger a
disapproval -- however, a single disapproval can be for
many multiple issues (many having nothing to do with
any image issue or with the feed itself).
unfortunately, forum members cannot look into feeds or accounts --
if you would like to post more exact details here in the public forum
we may be able to offer more exact suggestions or possibilities.
google may sometimes take 24-72 hours or so before all items
in a submitted feed are inspected and have a quiescent status --
and the website has been crawled for any quality related issues.
an email from google should have been sent
regarding the disapproval issue -- otherwise,
google may be contacted directly here --
as to the robots.txt specifically, regardless of what might seem well
or might have worked in the past (or until the issue is resolved) try
adding the following nine records (lines) to the very end of the site's
Re: Google Products Disapproved or Invalid Robots.txt & images ok?
After much hair pulling I have just discovered that all the disapproved images in my google feed are due to the fact that I copied and pasted my site image URLs into the feed using HTTPS instead of HTTP.
I know this mistake is elementary to the point of being stupid, but it is so obvious a mistake that I simply did not see it . Added to which all the images in my feed with the incorrect HTTPS were all valid images but not crawable due to the incorrect prefix.
I spent many fun filled hours trying to understand why my Robts.txt file was deficient when it technically was not. It was the image URL that was not correct.
It is a great pity that no Google documuntation nor advice from Adwords support ever mentioned this as a possible error.
Maybe this "discovery" will help others.