Leakage or dodgy anonymous scraping by Google
We maintain 1000s of records in Google Merchant Centre so that the products we sell on our ecommerce store can be advertised as Shopping Ads in Adwords.
For weeks now, we have had our website scraped by anonymous visitor(s) landing on our product landing pages. No other pages are visited, only the urls provided to Merchant Centre are being hit, thousands upon thousands each day.These visits are not coming from Googlebots of any kind, the user agents and referer strings are clearly being faked or blanked. Also, the IP addresses are not in Google's ranges but are either Amazon EC2 or TOR IPs.
As a test we added some impossible-to-guess coding to the URLs sent to Merchant Centre and within hours these visits started to include that coding.
Either Google are doing this scraping themselves (above and beyond their normal bot activity) or are selling this Merchant Centre data to 3rd parties who are then scraping using the URLs.
We are blocking these visits, but they don't stop! (still in the logs). Is this something to worry about or should this be reported somehow (if so, how?). As these are not Googlebot visits, presumably it is safe to continue to block them?
Re: Leakage or dodgy anonymous scraping by Google[ Edited ]
July 2015 - last edited July 2015
for merchant-center related crawl issues google may be contacted directly --
also, the webmaster-forums may be consulted for crawling related issues --