Fill in the options below to generate a valid robots.txt file. Everything runs in your browser — nothing you enter here is sent to our server.
Computed entirely in your browser — nothing you enter here is sent to our server.
Computed entirely in your browser — nothing you enter here is sent to our server.
Generating…
How This Tool Works
robots.txt tells search engine crawlers which parts of your site they may or may not crawl. This tool builds a standard robots.txt: choose whether to allow crawling by default, list any specific paths to disallow, optionally set a crawl-delay, and optionally reference your sitemap. Save the result as robots.txt in your site’s root directory.
robots.txt controls crawling, not indexing — a disallowed page can still appear in search results (e.g. linked from elsewhere) unless you also use a noindex meta tag. Blocking crawling and preventing indexing are two different things.
FAQ
Does disallowing a page in robots.txt remove it from search results?
Not necessarily — robots.txt only asks crawlers not to fetch that page. If the URL is discovered another way (like a link from another site), it can still be indexed without its content being crawled. Use a noindex meta tag for reliably keeping a page out of search results.
Where does robots.txt go?
In your site’s root directory, so it’s reachable at yoursite.com/robots.txt — that’s the only location crawlers check.
Is my data uploaded anywhere?
No. The file is generated entirely in your browser. Nothing you enter here is sent to our server.