Found this interesting little website that explains the things that should/could go in your <head>: https://github.com/joshbuchea/HEAD
What do you guys think about this bit? Is it important? Unnecessary?
Control the behavior of search engine crawling and indexing
meta name="robots" content="index,follow,noodp" - All Search Engines
meta name="googlebot" content="index,follow" - Google Specific
Re: Crawling[ Edited ]
June 2016 - last edited June 2016
@James W, Github is a fun site with a lot of really smart information.
As for these specific meta tags, yes, they can be added... but not every site will use every tag.
Index - add all content.
Follow - come back and slurp the data again.
Noodp - no open directory project.
The last ones directs the bot to not include info from DMOZ and sources such as that. Their data could be outdated and so you are telling the bot to not use them as a resource.
Side Note: remember to always submit an XML Sitemap in Google Search Console to direct Google when to revisit the site, what pages are contained in your site, when the pages were last updated (change the date every time you make a change), and the priority of the page. You can learn more about Sitemaps here.
There are many bot commands you can use but only some will be adhered to and only some commands by some bots.
This resource by Google will help you learn more about the uses.
Happy Tuesday and be Google great!
Google AdWords Top Contributor | Google Partner | GYBO | Local Guide | My Profile