Partners
3.3K members online now
3.3K members online now
Community space to get to know one another, talk about marketing and advertising strategy
Guide Me
star_border
Reply

Crawling

Follower ✭ ☆ ☆
# 1
Follower ✭ ☆ ☆

Found this interesting little website that explains the things that should/could go in your <head>: https://github.com/joshbuchea/HEAD

 

What do you guys think about this bit? Is it important? Unnecessary?

 

Control the behavior of search engine crawling and indexing

meta name="robots" content="index,follow,noodp" - All Search Engines

meta name="googlebot" content="index,follow" - Google Specific 

1 Expert replyverified_user

Re: Crawling

[ Edited ]
Top Contributor
# 2
Top Contributor

@James W, Github is a fun site with a lot of really smart information. 

 

As for these specific meta tags, yes, they can be added... but not every site will use every tag. 

 

Index - add all content.

Follow - come back and slurp the data again.

Noodp - no open directory project.

 

The last ones directs the bot to not include info from DMOZ and sources such as that. Their data could be outdated and so you are telling the bot to not use them as a resource.

 

Side Note: remember to always submit an XML Sitemap in Google Search Console to direct Google when to revisit the site, what pages are contained in your site, when the pages were last updated (change the date every time you make a change), and the priority of the page. You can learn more about Sitemaps here.  

 

There are many bot commands you can use but only some will be adhered to and only some commands by some bots. 

 

This resource by Google will help you learn more about the uses.

 

Happy Tuesday and be Google great!

_________________________________________________________
Google AdWords Top Contributor | Google Partner | GYBO | Local Guide | My Profile