AdWords is now Google Ads. Our new name reflects the full range of advertising options we offer across Search, Display, YouTube, and more. Learn more

Ads
1.7K members online now
1.7K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

Adsbot-Google vs. Googlebot - PLA landing page verification

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

I'm trying to confirm what I believe is conflicting and/or misleading information from Google on allowing their crawler(s) to access URLs to utilize in PLA and other ads.

 

This article says to allow Googlebot and Googlebot-image access to product pages used in the shopping feed:

https://support.google.com/merchants/answer/6098185?hl=en#

 

Allowing Googlebot (generic crawler) to access pages may not be good in some SEO scenarios. For instance, allowing unfriendly URLs to be crawled could lead to duplicate pages/content issues.

 

There is a specific bot for Ads (Adsbot-Google), which is referenced here: 

https://support.google.com/webmasters/answer/1061943?hl=en

 

A further description of how Adsbot is used to evaluate landing page scoring, etc is here:

https://support.google.com/adwords/answer/2404197?hl=en

 

Does "Googlebot" need access to PLA or AdWords landing pages at all or is Adsbot-Google the appropriate user-agent to crawl/verify product (landing) pages?

 

Hopefully I'm not crossing services/ad platforms with different requirements. Assuming AdWords, PLA/Shoppping, etc. use the same bot.

 

Any help/confirmation is appreciated. Thanks in advance.

1 Expert replyverified_user
1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Cory H
September 2015

Re: Adsbot-Google vs. Googlebot - PLA landing page verification

[ Edited ]
Top Contributor
# 2
Top Contributor


currently, product-listing-ads require all three
user-agents access to the website and images --
googlebot, googlebot-image, and adsbot-google.

the crawlers are used primarily to determine adherence to all merchant-center
rules, requirements, and policies with respect to the website, landing-page, and
all submitted images, and to crawl and cache images -- any of those agents may
also be used for scheduled-fetch, automatic-item-updates, and similar features.

the site and images must be accessible to all google's
crawl requests and must respond properly, at all times --
google may crawl a site at any time.

the common user-agent for organic-search has been true
for quite some time -- generally, to prevent indexing, simply
allow google to crawl the page but use a noindex meta-tag.

see also
https://support.google.com/webmasters/answer/93710

 

View solution in original post

Marked as Best Answer.
Solution
Accepted by topic author Cory H
September 2015

Re: Adsbot-Google vs. Googlebot - PLA landing page verification

[ Edited ]
Top Contributor
# 2
Top Contributor


currently, product-listing-ads require all three
user-agents access to the website and images --
googlebot, googlebot-image, and adsbot-google.

the crawlers are used primarily to determine adherence to all merchant-center
rules, requirements, and policies with respect to the website, landing-page, and
all submitted images, and to crawl and cache images -- any of those agents may
also be used for scheduled-fetch, automatic-item-updates, and similar features.

the site and images must be accessible to all google's
crawl requests and must respond properly, at all times --
google may crawl a site at any time.

the common user-agent for organic-search has been true
for quite some time -- generally, to prevent indexing, simply
allow google to crawl the page but use a noindex meta-tag.

see also
https://support.google.com/webmasters/answer/93710

 

Re: Adsbot-Google vs. Googlebot - PLA landing page verification

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭
Thanks for the quick feedback.

The noindex is a feature we were aware of, but the client has been unable to support in the situation that led to my inquiry.

We'll certainly look to reiterate that directive to the client as they continue to use the undesired URL format for the PLA listings.

Re: Adsbot-Google vs. Googlebot - PLA landing page verification

[ Edited ]
Top Contributor
# 4
Top Contributor


first, you're welcome.

blocking crawling does not necessarily block indexing,
and is a rather common misnomer -- there are many
many ways a page can be indexed but never crawled;
for example, a link from another site to the page.

crawling and indexing are mostly disjoint.

about the only way to stop a page from ever being indexed
is to use password-protected sever-side-authentication,
before the first time the page is ever added to a domain --
which is not allowed for product-listing-ads pages.

as to being unable to use noindex, depending on the
details of the undesired urls, canonical might be used --
https://support.google.com/webmasters/answer/139066

however, these are mostly off-topics for this forum;
the webmaster-central-help-forum is more appropriate --
https://productforums.google.com/forum/#!forum/webmasters

Re: Adsbot-Google vs. Googlebot - PLA landing page verification

[ Edited ]
Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭

Thanks again for going above & beyond...

So the pages do/have a canonical tag on them and Google isn't utilizing it. We've been seeing this a lot lately on many other sites. Unless 100% of signals are aligned, including the canonical, Google isn't trusting and/or utilizing them as you'd expect. They are essentially keeping the URLs in the index.

Not really looking for further SEO tips, but I appreciate everything so far.

I believe you've addressed our concerns with the bots specific to PLA ads.