How to customize robots.txt file on blogger

Blogger (Blogspot) allows you to customize your robots.txt file, which provides instructions for search engines like Googlebot on how to crawl and index your blog. Here's a breakdown of typical settings and an example:

Default Settings:

By default, Blogger includes basic robots.txt instructions that look something like this:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /
Sitemap: http://[YOUR_BLOG_URL]/feeds/posts/default?orderby=updated

Explanation of Default Settings:

• User-agent: Mediapartners-Google: This line specifies that the following rule applies to the Google Adsense crawler.
• Disallow: This indicates that the crawler should not access anything after the forward slash (/). By default, this likely refers to internal search functionality within Blogger itself (not something you typically want indexed).
• *User-agent: : This line specifies that the following rule applies to all search engine crawlers (including Googlebot).
• Disallow: /search: This instructs crawlers not to index your blog's search function results.
• Allow: /: This allows crawlers to index all content at the root level of your blog (/) and subdirectories.
• Sitemap: http://[YOUR_BLOG_URL]/feeds/posts/default?orderby=updated: This line provides the location of your blog's sitemap, which helps search engines understand the structure and content of your blog.

Customizing robots.txt in Blogger:

While the default settings are generally sufficient, you can customize your robots.txt file in Blogger to achieve specific goals:

1. Login to Blogger: Access your Blogger dashboard.
2. Settings: Click on the "Settings" option.
3. Search preferences: Scroll down to the "Crawling and indexing" section.
4. Enable custom robots.txt: Look for the option to "Enable custom robots.txt" and check the box.
5. Edit robots.txt: A text box will appear where you can edit the robots.txt content.

Example with Additional Disallow:

Let's say you have a specific folder on your blog named "/private" that contains content you don't want indexed. You can add a "Disallow" directive to block crawlers from accessing that folder:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: /private/  # Block access to /private folder
Allow: /
Sitemap: http://[YOUR_BLOG_URL]/feeds/posts/default?orderby=updated

Important Considerations:

• Be cautious when modifying robots.txt. Blocking important content can prevent search engines from indexing your blog effectively.
• Use clear and concise language in your robots.txt directives.
• Test any changes using the Robots.txt Testing Tool in Google Search Console to ensure Google understands your instructions.

By understanding the default settings and how to customize them, you can leverage robots.txt to optimize your Blogger website for search engine crawling and indexing.

Share :