AdWords
1.5K members online now
1.5K members online now
For questions related to Google Shopping and Merchant Center. Learn to optimize your Shopping ads
Guide Me
star_border
Reply

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

[ Edited ]
Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

We have a feed that has been working for several years. We made no changes on our end. It appears that in February, around the 9th, Google stopped fetching our feed. Merchant Centers says "No items processed". Underneath there is an error which reads "Failed to connect to the remote server. Please make sure the source URL in your feed configuration is a valid one".

 

The URL is valid, and I can load it and see the XML feed just fine in a browser. Status code is 200, nothing is cut off.

 

How do I get more details as to what the problem is?

2 Expert replyverified_user
1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Nick R
April 2017

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Visitor ✭ ✭ ✭
# 20
Visitor ✭ ✭ ✭

Thanks for the info and the help everyone. I've identified the problem:

 

The IPs Google fetches with use the user agent "Googlebot", but they don't have valid Reverse DNS records that resolve to googlebot.com or google.com. (There are no RDNS records on those IPs at all).

Due to the prevalence of spammers and other malicious users pretending to be Google, Google gave the following instructions to verify it's authenticity: https://support.google.com/webmasters/answer/80553?hl=en

Following this advise, we automatically blacklist IP addresses if they 1. claim to be `GoogleBot` and 2. do not have a RDNS record that are googlebot.com or google.com, per the documentation. As a result, the feed crawl IPs were being blacklisted one at a time, because they didn't pass the verification test.
 
I white-listed the IPs that I could find and the feed started working again.

I've also been told (by AdWords support after reporting the issue) that the Google team is working to add the correct RDNS records to IPs claiming to be `Googlebot` so they don't get flagged as bad traffic.
 
I've also been told that `google-xrawler` shouldn't be used to fetch the feed since the feed fetcher should only be identifying as `Googlebot`. I've noticed that there aren't any more `google-xrawler` entries yesterday or today (since white-listing and reporting the problem), only the expected `Googlebot`. (@Celebird: did the `google-xrawler` entries go away for you too?)
 
Hopefully this helps somebody!
 
 
P.S. I'm also sharing this handy command for searching all logs (including zipped logs) for all file accesses on a certain date:
 
zgrep -r 'google-merchant-feed.xml' /var/log | grep 2017-04-18



View solution in original post

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Visitor ✭ ✭ ✭
# 2
Visitor ✭ ✭ ✭

Having EXACTLY the same problem!

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Rising Star
# 3
Rising Star

Check that the data feed can be accessed from all countries and all ip's

Check that there is no encoding issue, certain characters can make the file corrupt.

 

I recommend checking the above.

 

Hope it helps.

Twitter | Linkedin | Community Profile | Shopping Feed Tips From FeedArmy
Did you find any helpful responses or answers to your query? If yes, please click on ‘Accept As Solution’

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Visitor ✭ ✭ ✭
# 4
Visitor ✭ ✭ ✭

Shortly after I posted this (on Friday at 8AM), it did fetch all of the items correctly, once. (I didn't make any changes on my end, so I'm assuming someone from Google read this thread and investigated). Prior, it hadn't done a proper fetch in months, possibly since the end of December.

 

Today (Sunday, 2 days later), it's back to "No items processed" and "Failed to connect to the remote server".

 

There are no IP address restrictions. Feed loads fine in a browser. I dropped the firewall completely and the "fetch now" button still doesn't fetch.

 

So whatever Google did on Friday got the items loaded for now, but it's still not fetching so these items are just going to expire again (and have data that's out of date).

I don't think this is a problem on my end, and if it is, I'm at the limits of what I can do to resolve it without a more detailed error message.

 

 

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Rising Star
# 5
Rising Star

When Google can not fetch, it will always be an issue on your side. If it did fetch and than did not again, than there is an issue with your server not being able to handle your data feed. For example if you add more and more products, you need more and more ram to process the file. Certain servers are not capable of handling large files.

 

Get in touch with your web developer to do some server and file testing, to ensure you have enough memory.

 

Also check you have enough data base memory

 

hope it helps.

Twitter | Linkedin | Community Profile | Shopping Feed Tips From FeedArmy
Did you find any helpful responses or answers to your query? If yes, please click on ‘Accept As Solution’

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Visitor ✭ ✭ ✭
# 6
Visitor ✭ ✭ ✭

There is no issue with the server or the feed. If there was not enough ram, I would not be able to view the feed in the browser. If the server was not responding, I would not be able to open it in the browser. How can I open a ticket with Google?

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

[ Edited ]
Top Contributor
# 7
Top Contributor

(1) generally, a browser cannot be used to verify data-feed fetches by google --
most browsers, by default, cannot effectively verify the automated googlebot,

adsbot-google, or googlebot-image crawls or related feed-file fetches.

 

that said, this is mainly a peer-to-peer forum -- forum-members cannot look
into any submitted feed or account details; forum-members can mainly offer
suggestions based on the details posted here in public.

posting more specific details, such as a specific url being used for the
scheduled-fetch, screen-captures from the account, log files from the
servers on which the file is hosted, etc., here within the public-forums,
may sometimes help others offer more specific suggestions.

(2) a ticket may be opened with google directly here --
https://support.google.com/merchants/contactflow

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Visitor ✭ ✭ ✭
# 8
Visitor ✭ ✭ ✭

I'd be willing to PM someone the feed URL if they wanted it, but I don't think posting it publicly would be a good idea. I checked the error and access logs for Nginx and I'm not seeing any request from Google for the feed URL. (or any request from Google other than Googlebot crawling of pages). When I open the feed in the browser, I do see an entry for me in the access log, so logging is working and the feed has no errors loading.

 

I also tried tailing syslog while making a "fetch now" request. There are a few iptables denied messages, but they appear to all be bad bots that should be blocked: 182.92.82.179, 182.180.107.131, 216.218.206.103, 211.217.90.121.

 

So If Google were blocked by the firewall, it would show up here, but no legitimate IPs are. It still couldn't fetch with the firewall disabled per my test last Sunday, so not a firewall issue.

 

Also, nothing interesting in auth.log.

 

Aside from these logs, are there other logs you might recommend I try?

 

It would appear that Google isn't even trying to connect to the server at all, or the request would be logged. I don't know what else I can do on my end. And I would re-iterate that we haven't made any changes to the server or feed scripts, and the feed has been working fine for several years now (until this issue stated).

 

I submitted a ticket to Google on Sunday through their online form. No reply or (even acknowledgement) as of yet.

Re: Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

[ Edited ]
Top Contributor
# 9
Top Contributor

first, thank you for the comprehensive reply.

unfortunately, private-messages and any similar off-line
communications are forbidden by the forum-guidelines.

normally, a scheduled-fetch triggers a get-request by
google and is indicated by a server log-entry from one

of google's user-agents via one of google's ubiquitous

worldwide servers/crawlers;
e.g.
173.194.90.35 - - [14/Mar/2017:22:23:29 -0400] "GET /help-forum/help-forum-au.xml HTTP/1.1" 200 - - "google-xrawler" "celebird-support.appspot.com" ms=7 cpu_ms=0 cpm_usd=0 loading_request=0 instance=- app_engine_release=1.9.48 trace_id=fa5020116f2d4de10e4724f8fcbaa22f

 

such details is one reason why most browsers simply

cannot effectively verify google's automated requests --

network/server configurations may block such requests,

but not from a typical browser/user-agent.

a similar entry should occur within the server's logs,
a few seconds or so after a fetch-now is requested,
or soon after a scheduled-time is reached.

 

if a different protocol is used (e.g. ftp, sftp) then the logs may

be under a separate area, depending on the hosting-company.

 

also, depending on the hosting-company, the (nginx/http/ftp) servers may

have different error, access, conditional, or syslog logs -- these should all
likely be checked; asking the hosting-company's support organization
directly may sometimes help.

scheduled-fetch details are also available when using a test-feed --
which may be a good option when attempting to locate a root-cause,
without adversely affecting a live/standard feed or live/active products.

google also has chat and phone support-specialists --
normal business-hours during weekdays are usually best;
indicating that the issues relate to shopping-ads and more
specifically to a merchant-center-account may also help.

see also
https://support.google.com/merchants/answer/1188998

Google stopped fetching feed in February, claims remote connection error, but feed loads fine.

Visitor ✭ ✭ ✭
# 10
Visitor ✭ ✭ ✭

https is the protocol. This is a VPS, so the hosting company only maintains the hardware and network layers. The log I checked does log all errors and requests for the domain in question, and I can see other Google entries just fine. Bing Ads uses the same feed and is fetching it just fine (and Bing Ads are buggy as you know what). I guess I'll have to chat or call.