How To Add Custom Robots.Txt File In Blogger (Blogspot)?

For better performance of our blog, we need to optimize it for better SEO. A better SEO is also needed to get organic traffic from search engines. One great step towards a better SEO is adding robots.txt file to your blog. It tells the search engine crawlers about what page to crawl and what not to. In Blogger (Blogspot) we have the option to customize the robots.txt file according to our needs. But before adding it to your blog firstly as a beginner we need to know about what is robots.txt file. So let's start the tutorial.

What is Robots.txt?

Robots.txt is a file that tells the search engine crawlers what page to crawl and what not to. Simply, you can exempt your blog demo pages, label pages, archive pages and others which you don't consider as important. Search engine crawlers always scan the robots.txt file before crawling any blog.

So let's start adding this great SEO improver to your blog.

Also Check : Custom Robots Header Tags Settings For Blogger (Blogspot)

Adding Robots.txt to Blogger (Blogspot) Blogs:

To add Custom Robots.txt file to your blog, follow the basic steps mentioned below:

STEP 1. Go to your blogger blog.

STEP 2. Navigate to Settings >> Search Preferences ››

STEP 3. Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes

STEP 4. Now paste the below robots.txt file code in the box.

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

STEP 5. Remember to change the highlighted Url (example.blogspot.com) with your blog address.

STEP 6. Click on the Save Changes button.

You are all done.


NOTE:

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

This code refers to the sitemap of our blog. By adding a sitemap link here we are simply optimizing our blog's crawling rate.

Means whenever the web crawlers scan our robots.txt file they will find a path to our sitemap where all the links of our published posts present.

Web crawlers will find it easy to crawl all of our posts.

Hence, there are better chances that web crawlers crawl all of our blog posts without ignoring a single one.

Also Check : How To Submit Blogger Sitemap To Google Search Console?

Note: This sitemap will only tell the web crawlers about the recent 25 posts. If you want to increase the number of links in your sitemap then replace the default sitemap with below one. It will work for the first 500 recent posts.

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If you have more than 500 published posts in your blog then you can use two sitemaps like below:

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

How to Check Your Robots.txt File? 

You can check this file on your blog by adding /robots.txt at the end of your blog URL in the web browser. For example:

http://www.yourblogurl.blogspot.com/robots.txt

Once you visit the robots.txt file URL you will see the entire code which you are using in your custom robots.txt file.

Final Words! 

This was today's complete tutorial on how to add custom robots.txt file in blogger.

I tried my best to make this tutorial as simple and informative as possible. But still, if you have any doubts or query then feel free to ask me in the comment section below.

Make sure not to put any code in your custom robots.txt settings without knowing about it. Simply ask me to resolve your queries. I'll tell you everything in detail.

Thanks, guys for reading this tutorial. If you liked it then please support me to spread my words by sharing this post on your social media profiles. Happy Blogging!

Tags

Blogger SEO