A robots.txt file gives crawl instructions to compliant search engine and automated agents. For most WordPress websites, a minimal file is safer than a long list of blocked folders. The generator keeps public theme, plugin, image, script, and stylesheet resources accessible so search engines can render pages correctly.
Build a minimal WordPress robots.txt file. Blocking a URL here does not guarantee that it will disappear from search.
How this tool helps
Robots.txt controls crawling, not guaranteed indexing or security. Never use it to protect confidential files because the file itself is public. Use authentication, permissions, noindex controls, and server rules for sensitive content.
How to use it
- Enter the website address.
- Choose whether to block admin, internal search, feeds, or selected AI crawlers.
- Include the WordPress sitemap when it is active.
- Publish the file at the website root and test it in the relevant search console.
Frequently asked questions
Should wp-content be blocked?
No. Search engines need access to public images, CSS, and scripts to understand and render pages.
Does Disallow remove a page from Google?
Not necessarily. A blocked URL can still appear when other pages link to it. Use an appropriate noindex method when removal is required.
Where should robots.txt be placed?
It should be available at the root, such as https://example.com/robots.txt.
Important: Review generated configuration before using it on a production website and confirm detector results against the website you are researching.