AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
2.7K members online now
2.7K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

Google Shopping Feed taking over Googlebot

[ Edited ]
Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

We offer a very large number of images printed on custom products and have a monthly main Google Shopping feed of around 1,200,000 items

 

As Googlebot seems to give priority to the shopping feed we are seeing very little of the site indexed, almost all that is crawled are the shopping pages

 

Individual product pages are no-indexed to reduce duplication penalties, only the main generic page for each image is indexed

 

What is the best way of getting the rest of the site crawled so it can be indexed please?

 

Google Shopping has been good to us and we don't want to turn it off to allow Googlebot to crawl the rest of the site.

 

Kindest thanks

 

Derek

1 Expert replyverified_user
1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Derek M
February 2016

Re: Google Shopping Feed taking over Googlebot

Top Contributor
# 4
Top Contributor
first, you're welcome.

if the traffic from googlebot is mainly from google-shopping
then, about the only way to reduce such traffic is to limit
the number of products being submitted -- for example,
only submit best-selling in-stock items.

there is no requirement to submit all in stock inventory items --
especially if the server cannot handle both google-shopping's
googlebot traffic and google-search's googlebot traffic.

otherwise, googlebot for google-shopping and googlebot for
google-search indexing have nothing whatever to do with one
another -- these are separate crawls for different purposes.

otherwise, as was indicated, for organic-google-search and google-search
indexing related issues the best likely course would be to simply ask the
question over in the webmaster-central-help-forum, which is currently here:
https://productforums.google.com/forum/#!forum/webmasters

View solution in original post

Re: Google Shopping Feed taking over Googlebot

[ Edited ]
Top Contributor
# 2
Top Contributor

the various user-agents related to google-shopping crawling --
such as googlebot, googlebot-image, adsbot-google -- are used

mainly to verify that submitted images, item data, landing-pages,
and the website in general, conform to all the rules and policies
required by product-listing-ads and cannot be blocked.

google-search related googlebot and indexing issues are

best addressed by the webmaster-central-help-forums --
https://productforums.google.com/forum/#!forum/webmasters

as an aside, generally, only items that are kept physically in stock
by the merchant should be submitted as product-listing-ads -- and is
otherwise a policy violation and grounds for disapproval or suspension
from the program, at any time.

Re: Google Shopping Feed taking over Googlebot

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭
Thank you for your feedback. We are aware of why the shopping feed has a high priority, and our products are agreed with Google Shopping. We have no desire to block Googlebot, quite the reverse.

The issue is that 99% of Googlebot traffic is pure Google Shopping feed, leaving our main site largely un-indexed or outdated.
Marked as Best Answer.
Solution
Accepted by topic author Derek M
February 2016

Re: Google Shopping Feed taking over Googlebot

Top Contributor
# 4
Top Contributor
first, you're welcome.

if the traffic from googlebot is mainly from google-shopping
then, about the only way to reduce such traffic is to limit
the number of products being submitted -- for example,
only submit best-selling in-stock items.

there is no requirement to submit all in stock inventory items --
especially if the server cannot handle both google-shopping's
googlebot traffic and google-search's googlebot traffic.

otherwise, googlebot for google-shopping and googlebot for
google-search indexing have nothing whatever to do with one
another -- these are separate crawls for different purposes.

otherwise, as was indicated, for organic-google-search and google-search
indexing related issues the best likely course would be to simply ask the
question over in the webmaster-central-help-forum, which is currently here:
https://productforums.google.com/forum/#!forum/webmasters

Re: Google Shopping Feed taking over Googlebot

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭
Ah ok thank you for your kind response. We can handle the traffic ok its simply that we would like Google to crawl the site faster to get rid of duplicates etc (which were removed many months ago) and its taking an age.

Our Directors were asking why its taking so long when Google is crawling many thousands of pages per day, at least we can now prove its the Google Product feed that is crawling and they are refusing to make that smaller, not that it keeps them quiet lol.

Kindest thanks

Derek

Re: Google Shopping Feed taking over Googlebot

[ Edited ]
Top Contributor
# 6
Top Contributor

first, you're welcome.

faster crawls and reducing googlebot
traffic are slightly different issues.

both can be helped by reducing the number of items submitted.

 

normally, there should not be any duplicates to remove with

respect to google-shopping, since submitting duplicates is a

policy violation.

for faster crawls, be certain that all id values
and all image url's submitted remain static --
but any change to the feed can slow googlebot.

otherwise, the proper use of rich-snippets and
inventory-update-feeds may sometimes help.

otherwise, replacing all feeds with the api may sometimes help --
as this can stagger and better control items being submitted
based on physical-inventory changes or changes on the site.

importantly, be certain that all items submitted as in stock
are physically in the merchant's on-hand inventory and also
matches the website's link landing-page details, exactly, at
all times, to help prevent a future disapproval or suspension.

 

if duplicates refer to organic google-search content

then, the best likely course is still to ask over in the

webmaster-help-forums -- or submitting a comment

within webmater-tools (search-console), under the

crawl-related section, directly to google.