Guide to file Robots.txt: what it is and why it is so important
In this article, we will explore the role of robots.txt, a small file that can make the difference between scoring a high ranking and languishing on the lowest deeps of the SERP.
What is robots.txt
Robots.txt’s role is to tell the crawler which pages it can require from your site. Beware, the spider can still see them. It just does not scan them. If you want to hide a page, you should rely on noindex instructions, as specified by Google’s Search Console Guide.
So, why do you need a robots.txt file? Because you can make crawling faster and smoother, saving your server from too many crawler requests. You can exclude duplicate or not essential pages from the scanning that can hurt your ranking.
Where to put robots.txt
You have to put the robots.txt file inside your website’s main directory so that its URL is http://www.mywebsite.com/robots.txt.
Do not put it elsewhere, or it won’t work.
How to create robots.txt file
Create a .txt file in your website’s main directory and call it “robots”. Remember that you can have only one robots.txt file per site.
Create a group
Create your first group. A robots.txt file can have one or more groups.
Each group has one or more instructions (also called rules). Remember to use only one instruction for each line.
Robots.txt istructions
Instructions can be of three types:
- user-agent: the crawler to which the rule applies.
- allow: all the files or directories to which the user-agent can access.
- disallow: all the files or directories to which the user-agent cannot access.
A rule must include one or more (or all!) user agents, and at least an allow or disallow instruction (or both).
Robots.txt examples
For example, to prevent Googlebot to scan your entire website, you must write in your robots.txt file something like:
#Prevent GoogleBot from scanning. (this is a comment. You can write what you want)
User-agent: googlebot
Disallow: /
If you want instead to exclude more than a directory for all crawlers:
User-agent: *
Disallow: /directory 1
Disallow:/ directory 2
(the asterisk means “all”)
Or maybe exclude all directories but one to a specific crawler:
User-agent: specific-crawler
Allow: /directory1 User-agent: * Allow: /
In this way, you’re stating that every other crawler can access to the entire website.
Finally, we can prevent the scanning of a specific file format, for example, jpg images.
User-agent: *
Disallow: /*.jpg$
The $ character establish a rule valid for all strings that end with .jpg.
To see more examples, visit the Google’s Search Console guide.
Learn more about Technical SEO
Technical SEO is not easy. But it’s fundamental to make SEO the right way.
Learn it reading our Guide for Beginners to Technical SEO.
