AdWords
5.2K members online now
5.2K members online now
Get started with AdWords - learn the basics to get set up for success
Guide Me
star_border
Reply

Help needed with Google crawlers being blocked by Wordfence

[ Edited ]
Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

A client of mine hired a SEO company to help promote her website. I believe they started a Google AdWords "Pay-Per-Click" campaign but tell me the campaign can't get started due to Google crawlers being blocked by Wordfence, which is software I run to keep hackers at bay.

In the Wordfence logs, I don't see any evidence of Google crawlers being blocked. The SEO company isn't able to provide any specifics regarding the error codes they are getting back which tell them the ad campaign isn't running.

What's the best way to proceed to determine why the ad campaign isn't running and if it's because a Google crawler is being blocked, how can I find out which IP addresses or hosts would be used to crawl the site? I see Googlebot crawling the site regularly and without any issues, so I'm a bit perplexed as to what the issue might actually be.

Thanks in advance for any assistance or advice you can provide!

Peace...

Tom

1 Expert replyverified_user
1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Tom D
January 2017

Re: Help needed with Google crawlers being blocked by Wordfence

Top Contributor
# 2
Top Contributor

assuming the root-cause is verified by email from google,
such issues are typically due to blocking one or another
country, country-specific sites, or related ip-ranges --
that is usually not allowed.

typically, for google to properly crawl the website, from all locales
and all crawlers, such blocking should be isolated to the back-end
authentication areas of the site, not to any areas google may crawl.

generally, the same website-content must be presented
to all users, within all locations, regardless of user-agent,

or any other user-specific or country-specific information.

the best likely course would be to verify the issue
with google and contact wordfence-support directly --

to be certain that everyone, from around the world,

can access the site.

see also
https://support.google.com/adwords/answer/2404197
https://support.google.com/adwords/answer/2456098
https://support.google.com/adwords/answer/1722040
https://support.google.com/adwordspolicy/answer/6020954
https://support.google.com/webmasters/answer/1061943

View solution in original post

Marked as Best Answer.
Solution
Accepted by topic author Tom D
January 2017

Re: Help needed with Google crawlers being blocked by Wordfence

Top Contributor
# 2
Top Contributor

assuming the root-cause is verified by email from google,
such issues are typically due to blocking one or another
country, country-specific sites, or related ip-ranges --
that is usually not allowed.

typically, for google to properly crawl the website, from all locales
and all crawlers, such blocking should be isolated to the back-end
authentication areas of the site, not to any areas google may crawl.

generally, the same website-content must be presented
to all users, within all locations, regardless of user-agent,

or any other user-specific or country-specific information.

the best likely course would be to verify the issue
with google and contact wordfence-support directly --

to be certain that everyone, from around the world,

can access the site.

see also
https://support.google.com/adwords/answer/2404197
https://support.google.com/adwords/answer/2456098
https://support.google.com/adwords/answer/1722040
https://support.google.com/adwordspolicy/answer/6020954
https://support.google.com/webmasters/answer/1061943

Help needed with Google crawlers being blocked by Wordfence

Participant ✭ ✭ ✭
# 3
Participant ✭ ✭ ✭

Some great info from @Celebird   To test how your site renders (or not) around the world, here is a site that I learned about from a Google support rep a while back:

 

https://geopeeker.com/

 

hth

Help needed with Google crawlers being blocked by Wordfence

Visitor ✭ ✭ ✭
# 4
Visitor ✭ ✭ ✭

Thanks for the information!  Generally speaking, the activity that gets blocked are attempts to access malicious URLs that Wordfence blocks automatically and suspicious activity I see from various IP addresses.  For example, a log entry might contain a URL with commands intended to be run by Wordpress to do who knows what.   The site is a one page site, so it's pretty simple in structure.  I do have Wordfence log entries that show Googlebot being allowed to crawl the site.  Some systems try to pose as Google crawlers and I tend to block those.

 

A "fake" Google crawler being defined as a system reporting to be a Googlebot in its User Agent string but it's IP address either doesn't resolve to a hostname in Google's domain or it will resolve to some random ISP.  For, example a machine will report to be a Googlebot but the IP address will resolve to {blah}.comcast.net or something in some other country.

 

I'll read the other links you posted to see what information is contained therein. 

 

Thanks!

 

Peace...


Tom

Help needed with Google crawlers being blocked by Wordfence

Visitor ✭ ✭ ✭
# 5
Visitor ✭ ✭ ✭

Thanks for the link to GEO Peeker!  I'll check it out now!

 

Peace...


Tom

Help needed with Google crawlers being blocked by Wordfence

Top Contributor
# 6
Top Contributor

first, you're welcome.

 

typically, neither ip-addresses nor user-agents nor

random url parameters are allowed to be blocked;

the former two due to the policies, the later due to

how google handles ads and tracking.

 

googlebot is not the only user-agent used

and crawling is not the only way google

may inspect or verify site content for ads.

 

the best likely course would be to simply work

directly with google and wordfense support --

who should have other clients who purchase

advertising from google.

 

 

Help needed with Google crawlers being blocked by Wordfence

Visitor ✭ ✭ ✭
# 7
Visitor ✭ ✭ ✭

Thanks again for the information.  Contacting Google directly appears to be the biggest challenge for me, since I'm not sure how to contact them to get more information about what's preventing the campaign from running.  That's one reason I started this thread; to see what information I could get from the community, in addition to possibly finding out how to contact Google directly for support.

 

I presume the SEO company involved has access to some kind of Google AdWords account which is used for the Pay-Per-Click ad campaign. I'm in the process now of trying to get access to that dashboard so I can get more information about the ad campaign not running.

 

The IP addresses currently being blocked by me, manually, are usually for requests that seem unusual.  For example, a machine in India or the Ukraine will attempt to access the Wordpress login page and the IP address doesn't resolve to a hostname associated with Google or might resolve to a "TOR" host.  There is no link to the Wordpress login page, on the site, so that's not something a crawler would find anywhere on the page.

 

I guess what I'm saying is some of these URLs don't appear to have anything to do with the site content, itself.   One thing I did notice is there's a <meta> element for robots with "noindex,nofollow" specified.  That would tell Google (and other search engines) not to index the site and to follow any links on the site.   In the links you supplied above, I read descriptions about this but nothing that indicated its impact on AdWords.

 

In any event, what I'll do is contact Wordfence support and send them some of the URLs I'm seeing in the logs and confirm whether or not these are actually malicious or hacking attempts.

 

Thanks again!

 

Peace...

 

Tom

Help needed with Google crawlers being blocked by Wordfence

Visitor ✭ ✭ ✭
# 8
Visitor ✭ ✭ ✭

Thanks again for the replies.  It turns out, the reason the ad was not approved to run is due to some content on the site, not due to being blocked by Wordfence.  So, we're addressing that issue now.

 

Thanks for the help!

 

Peace...

 

Tom

Help needed with Google crawlers being blocked by Wordfence

Top Contributor
# 9
Top Contributor

first, you're welcome.

 

that sounds like good news and thank you for the update.