Robots.txt Generator
Generate custom robots.txt files to control search engine crawling and optimize site indexing.
Settings
Disallow: / rule.robots.txt
robots.txt to the
root of your website (e.g. https://example.com/robots.txt) so search engines can find
it.About Robots.txt Generator
A robots.txt file tells search engines and other crawlers which parts of your site they may visit. This generator builds one for you. Choose whether crawlers are allowed by default, list any paths to block (one per line), add your sitemap address, and optionally set a crawl delay. You can also block specific bots: Googlebot, Bingbot, GPTBot, Google-Extended, CCBot and AhrefsBot. Copy the result or download it as robots.txt.
Upload the file to the root of your site, so that it opens at yourdomain.com/robots.txt. Be careful with the settings. Blocking everything, or blocking Googlebot, stops Google from crawling your site and can remove your pages from search. Do not block the CSS and JavaScript files that your pages need to display.
Robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it does not hide pages from people, and a blocked page can still appear in search results if other sites link to it. To keep a page out of search, use a noindex tag instead. Google ignores the crawl-delay line, though some other crawlers read it. Blocking AI crawlers such as GPTBot or Google-Extended limits how your content is used for AI products, and it may also reduce how often AI assistants mention your site, so decide on purpose.
Who uses it and when
New sites
Create a basic file that points to your sitemap.
Private sections
Ask crawlers to stay out of admin or staging paths.
Bot control
Block specific crawlers you do not want.
Fixing mistakes
Replace a file that blocks too much.
Why use this Robots.txt Generator
A correct robots.txt in a minute.
Bot toggles
Block named crawlers with a switch.
Sitemap line
Adds your sitemap address after checking it is a valid URL.
Copy or download
Get robots.txt ready to upload.
How it works
Set the default
Allow or disallow crawlers overall.
Add paths, bots and sitemap
List blocked paths and your sitemap URL.
Upload the file
Place robots.txt in your site's root folder.
Frequently asked questions
Does robots.txt keep pages out of Google?
Not reliably. It blocks crawling, not indexing. Use a noindex tag to keep a page out of results.
Where does the file go?
In the root of your domain, so it opens at /robots.txt.
Does Google use the crawl delay?
No. Google ignores it, although some other crawlers respect it.
Should I block AI crawlers?
It is your choice. Blocking them limits use of your content for AI, and may reduce mentions in AI answers.
Guides and articles
- 14 Free Online Tools You’ll Actually Use Every Day in 2026 Most tool websites do the same thing: they promise "free" tools, then hit you with a sign-up wall, a watermark,…