Welcome Back

Enter your details to access your account.

OR
SEO Tools

Robots.txt Generator

Generate custom robots.txt files to control search engine crawling and optimize site indexing.

Free · No sign‑up Added Jun 2026

Settings

Paths are blocked for all robots. A leading slash is added automatically.
Crawl delay must be a whole number between 0 and 3600.
Enter a full URL starting with http:// or https:// — it will be added once valid.
Switch a bot on to fully block it with its own Disallow: / rule.
robots.txt
Upload the generated file as robots.txt to the root of your website (e.g. https://example.com/robots.txt) so search engines can find it.

About Robots.txt Generator

A robots.txt file tells search engines and other crawlers which parts of your site they may visit. This generator builds one for you. Choose whether crawlers are allowed by default, list any paths to block (one per line), add your sitemap address, and optionally set a crawl delay. You can also block specific bots: Googlebot, Bingbot, GPTBot, Google-Extended, CCBot and AhrefsBot. Copy the result or download it as robots.txt.

Upload the file to the root of your site, so that it opens at yourdomain.com/robots.txt. Be careful with the settings. Blocking everything, or blocking Googlebot, stops Google from crawling your site and can remove your pages from search. Do not block the CSS and JavaScript files that your pages need to display.

Robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it does not hide pages from people, and a blocked page can still appear in search results if other sites link to it. To keep a page out of search, use a noindex tag instead. Google ignores the crawl-delay line, though some other crawlers read it. Blocking AI crawlers such as GPTBot or Google-Extended limits how your content is used for AI products, and it may also reduce how often AI assistants mention your site, so decide on purpose.

Who uses it and when

New sites

Create a basic file that points to your sitemap.

Private sections

Ask crawlers to stay out of admin or staging paths.

Bot control

Block specific crawlers you do not want.

Fixing mistakes

Replace a file that blocks too much.

Why use this Robots.txt Generator

A correct robots.txt in a minute.

Bot toggles

Block named crawlers with a switch.

Sitemap line

Adds your sitemap address after checking it is a valid URL.

Copy or download

Get robots.txt ready to upload.

How it works

Set the default

Allow or disallow crawlers overall.

Add paths, bots and sitemap

List blocked paths and your sitemap URL.

Upload the file

Place robots.txt in your site's root folder.

Frequently asked questions

Does robots.txt keep pages out of Google?

Not reliably. It blocks crawling, not indexing. Use a noindex tag to keep a page out of results.

Where does the file go?

In the root of your domain, so it opens at /robots.txt.

Does Google use the crawl delay?

No. Google ignores it, although some other crawlers respect it.

Should I block AI crawlers?

It is your choice. Blocking them limits use of your content for AI, and may reduce mentions in AI answers.

Guides and articles