WordPress Robots.txt Generator
Your robots.txt file tells search engines which parts of your site to crawl. Current best practice for WordPress is to keep it lean: block only the admin area, never block the content folder, and point crawlers to your sitemap. This generator follows exactly that.
This follows current best practice: it blocks only the admin area, keeps admin-ajax open, and never blocks wp-content.
How to use it
- Enter the full address of your sitemap, usually ending in sitemap_index.xml or sitemap.xml.
- Tick any optional rules you want, such as blocking internal search results or AI training crawlers.
- Copy the result into a file named robots.txt in your site root folder.
Do not block wp-content. Your CSS, JavaScript, and images live there, and blocking them stops Google from rendering your pages correctly.
Frequently asked questions
Do I need a robots.txt file at all?
No. WordPress serves a virtual one by default. A custom file gives you more control and lets you add your sitemap.
Does robots.txt stop a page from appearing in search?
No. It controls crawling, not indexing. Use a noindex tag to keep a page out of results.