SEO
Optimising WordPress robots.txt for SEO
29 December 2024 · 6 min read
The robots.txt file is a simple text file that gives instructions to search engine robots about which pages they may and may not crawl. Although it is small, it has a big effect on how search engines treat your website.
At WP Maintainer we install and optimise robots.txt for all our clients. In this article we explain how to do this yourself.
What is robots.txt?
Robots.txt is located in the root of your website (example.com/robots.txt) and contains rules such as: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Standard WordPress robots.txt
WordPress automatically generates a robots.txt with: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
This is a good basis, but can be further optimised.
What should you exclude?
Exclude these pages to save crawl budget: **Admin area:** Disallow: /wp-admin/ Disallow: /wp-login.php
What should you NOT exclude?
Common mistakes: - **Entire folders with important content:** /wp-content/ contains themes, plugins and uploads - **CSS and JavaScript:** Google needs these to render the page correctly - **The sitemap:** This must be findable
Edit robots.txt in WordPress
Test your robots.txt
Use the Robots Testing Tool in Google Search Console to check whether: - Important pages are not accidentally blocked - Unwanted pages are correctly excluded - The sitemap is correctly mentioned
Common mistakes
- **Disallow: /** blocks the entire website - **Wrong syntax:** robots.txt is case-sensitive - **Not mentioning the sitemap:** This causes Google to miss the sitemap - **Relying on robots.txt for security:** It is not a security measure
WP Maintainer service
We configure and optimise your robots.txt as part of our SEO maintenance. We check whether search engines crawl the right pages and that no important content is blocked.