Basic robots.txt template for blogger

Here's a basic robots.txt template specifically designed for Blogger websites:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /
Sitemap: http://[YOUR_BLOG_URL]/feeds/posts/default?orderby=updated

Explanation of Directives:

• User-agent: Mediapartners-Google: This line specifies that the following rule applies to the Google Adsense crawler. "Disallow:" following it indicates the crawler shouldn't access anything specific after the forward slash (/).

• *User-agent: : This line specifies that the following rule applies to all search engine crawlers (including Googlebot).

• Disallow: /search: This instructs crawlers not to index your blog's search function results. Blog search results are typically dynamic and don't hold valuable content for search engines.

• Allow: /: This allows crawlers to index all content at the root level of your blog (/) and subdirectories. This ensures your blog posts and other relevant pages are accessible for indexing.

• Sitemap: http://[YOUR_BLOG_URL]/feeds/posts/default?orderby=updated: This line provides the location of your blog's sitemap, which helps search engines understand the structure and content of your blog. Blogger automatically generates a sitemap at the specified location.

Using this template in Blogger:

Blogger allows you to customize your robots.txt file directly within the platform. Here's how:

1. Login to your Blogger dashboard.
2. Click on "Settings".
3. Scroll down to the "Crawling and indexing" section under "Search preferences".
4. Look for the option to "Enable custom robots.txt" and check the box.
5. A text box will appear where you can paste the robots.txt template and customize it further if needed.

Remember:

• This is a basic template and you can adjust it based on your specific needs.
• Be cautious when adding additional "Disallow" directives, as you might accidentally block important content.
• Use clear and concise language in your directives.
• Regularly review and update your robots.txt file as your blog evolves.

By using this basic template and understanding its purpose, you can effectively manage how search engines crawl and index your Blogger website.

Share :