How to Create a robots.txt in 2026: Guide with Examples, AI Crawlers and a Mistake We Made Ourselves

Learning how to create a robots.txt takes two minutes. A text file, a few lines, upload it, done. That is what almost every guide says, and in principle it is true. For this article we did not just copy the syntax, though. We measured what actually happens to our own robots.txt on getmind.io: who fetches it, who obeys it and how different programs read it.

The result was more uncomfortable than expected. Our own robots.txt, just five lines long, was read by Python’s standard parser in exactly the opposite way Google reads it. Two directories we had blocked counted as allowed for every program using that parser. The cause was a single line that appears in countless templates online and that we had copied without thinking. The logs also showed something else: a blocked path in robots.txt is an invitation for attackers. Of 148 requests for a file inside our blocked folder, 118 came from IP addresses that had read the robots.txt first.

This guide shows you how to create a robots.txt that every crawler understands the same way, which templates make sense for WordPress, online shops and AI crawlers, where the limits are and which mistakes we made ourselves. All numbers come from our own server logs from October 9 to 19, 2026 (101,447 requests) and from tests run on October 19, 2026.

Amit Tiwari’s class (October 2025) walks through robots.txt, meta robots and X-Robots-Tag in one go, which is a good companion to this article. What it does not cover are the parser differences and the security angle we measured below.

What Is a robots.txt File?

A robots.txt is a plain text file in the root directory of your domain, always reachable at https://your-domain.com/robots.txt. It tells crawlers, the automated programs run by search engines, AI companies and SEO tools, which parts of your website they may fetch and which they may not.

The format is officially called the Robots Exclusion Protocol. It has existed since 1994 but was only an informal convention for almost 30 years. It was not standardized until September 2022, as RFC 9309, largely driven by Google. That standard matters because it settles questions every crawler used to answer differently. One of them caught us out, more on that shortly.

Three things you should know from the start:

1. robots.txt is a request, not a lock. Well-behaved crawlers follow it, but nothing is enforced. Any browser, script or attacker can still open a blocked URL.
2. It controls crawling, not indexing. A blocked page can still appear in Google if other pages link to it. It then shows up without a description (“No information is available for this page”).
3. It applies per host and protocol. https://www.example.com/robots.txt only governs www.example.com, not shop.example.com and not http://example.com.

Why Do You Need a robots.txt?

The honest answer first: a small website does not really need a robots.txt. If the file is missing and the server returns a clean 404, every crawler assumes it may fetch everything. That is usually exactly what a normal website wants.

A robots.txt becomes useful in four situations:

• Saving crawl budget: online shops with filters, sort orders and internal search pages quickly generate millions of URL variants. robots.txt keeps crawlers out of those endless spaces so they spend their time on your real product and category pages.
• Reducing server load: if aggressive SEO crawlers or AI bots hammer your database, you can block them specifically. That only works for those that follow rules, though.
• Controlling AI training: since 2023, robots.txt has been the main way to decide whether GPTBot, ClaudeBot, Google-Extended and others may use your content to train language models.
• Announcing your sitemap: the Sitemap: line is the simplest way to tell every search engine about your XML sitemap, including those where you have no account.

What robots.txt cannot do: remove pages from Google (you need noindex for that), protect confidential data (you need a password) or stop malicious bots (you need a firewall or rate limits).

Structure and Syntax: the Five Directives You Need

A robots.txt consists of groups. Each group starts with one or more User-agent lines followed by rules. That is all there is.

User-agent: Googlebot
Disallow: /internal/

User-agent: *
Disallow: /tmp/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

User-agent

Specifies which crawler the following rules apply to. * means “everyone without a group of their own”. The name is case-insensitive. Important: a crawler follows exactly one group, the most specific one that matches it. If Googlebot has its own group, it ignores the * group entirely. This is one of the most common mistakes: you block something under *, then add a Googlebot group that lacks the block.

Disallow

Forbids every URL whose path starts with the given value. Disallow: /admin also blocks /administrator, /admin-login and /admin.php. To block only the folder, write Disallow: /admin/. An empty Disallow: means “nothing is forbidden”.

The path is case-sensitive: Disallow: /admin does not block /Admin. We confirmed this in our test (see below).

Allow

Explicitly permits a path, usually as an exception inside a blocked area:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Allow was not part of the original convention but is officially supported since RFC 9309. How Allow and Disallow are weighed against each other is exactly where parsers diverge.

Sitemap

Points to your XML sitemap. The line belongs to no group and may appear anywhere, but it must contain a full URL (https://…, not /sitemap.xml). Multiple sitemap lines are allowed.

Wildcards: * and $

Google, Bing and RFC 9309 support two special characters in paths:

• * matches any sequence of characters: Disallow: /*?sort= blocks every URL with the sort parameter.
• $ marks the end of the URL: Disallow: /*.pdf$ blocks all PDFs, but not /file.pdf?version=2.

We checked this with an RFC-compliant parser: /a.pdf was blocked, /a.pdf?x=1 was allowed, exactly as expected. Python’s built-in parser, on the other hand, does not know wildcards at all and considered /a.pdf allowed as well.

What About Crawl-delay, Noindex and Host?

• Crawl-delay is not standardized. Google ignores it completely; Bing and Yandex respect it. If you want to slow Googlebot down, you have to do it through server responses (429/503).
• Noindex in robots.txt was officially retired by Google on September 1, 2019. It no longer works.
• Host was a Yandex extension and is meaningless today.

Our Test: Three Parsers, One File, Two Opinions

This is where it gets interesting. Until this morning, the robots.txt on getmind.io looked like this:

User-agent: *
Allow: /
Disallow: /tmp/
Disallow: /kaufland-preise/

Sitemap: https://getmind.io/sitemap-index.xml

Looks reasonable: allow everything in principle, block two internal folders. You will find this exact shape in countless templates. We ran the file through three different parsers and asked: may GPTBot fetch /tmp/x?

The difference lies in how a rule is selected. RFC 9309 and Google say: when several rules match, the longest, most specific one wins. Disallow: /tmp/ (5 characters) beats Allow: / (1 character). Python’s standard module uses the older principle “the first matching rule wins”. And the first rule was Allow: /, which matches every single URL. Everything after it is never consulted.

This is not a theoretical problem. urllib.robotparser sits inside a lot of small crawlers, scripts and home-grown scrapers, especially in the AI space where so many tools are written in Python. For all of those programs, our block was invisible.

As a cross-check we tested a second case where even the RFC parsers have to look closely: Allow: /page versus Disallow: /*.php for the URL /page.php. Both rules are the same length (five characters). According to RFC 9309, the least restrictive rule wins a tie, so Allow. Our Node parser still reported “blocked”. Even RFC-oriented implementations disagree on edge cases.

The Fix: Delete One Line

The solution was trivial. Allow: / is redundant, because anything not forbidden is allowed anyway. Without that line, all three parsers read the file identically:

User-agent: *
Disallow: /tmp/
Disallow: /kaufland-preise/

Sitemap: https://getmind.io/sitemap-index.xml

This version has been live on getmind.io since October 19, 2026. The lesson: write your robots.txt so it does not depend on the selection logic. In practice that means no blanket Allow: / lines, Allow only for genuine exceptions within a Disallow area, and those Allow exceptions always longer than the Disallow they belong to.

How to Create a robots.txt: Step by Step

Step 1: Check Whether One Already Exists

Open https://your-domain.com/robots.txt in your browser. Many systems generate one automatically. WordPress serves a virtual robots.txt that does not exist as a file on the server but is generated by PHP. Shopware, Shopify and many site builders do something similar. If you then upload a real file, it replaces the virtual version, which is worth knowing before you wonder why your Yoast settings stopped applying.

Also check the HTTP status code, not just the content. curl -I https://your-domain.com/robots.txt shows it immediately. Google treats status codes very differently:

The 5xx case is particularly treacherous: if your robots.txt returns a 500 for days because of a server problem, Google may stop crawling entirely. On getmind.io the file returns 200 with content-type: text/plain; charset=utf-8, and http:// and www. redirect cleanly to the HTTPS version via 308 and 301.

Step 2: Decide What to Block

Make a list with three columns: what should be crawled, what should not, and which crawlers you want to treat separately. Typical candidates for blocking:

• Internal search result pages (/search, /?s=)
• Filter and sort parameters in shops (?sort=, ?filter=)
• Cart, checkout, customer account
• Staging or test directories (these really belong behind a password, see below)

You should not block CSS and JavaScript files. Google renders pages like a browser. Without the stylesheets it sees a broken page and may judge it as not mobile-friendly.

Step 3: Write the File

Open a plain text editor (not Word!) and save the file as robots.txt with UTF-8 encoding. The file name must be lowercase. Google reads at most 500 KiB; anything beyond is ignored. For comparison: our file is 105 bytes, Wikipedia’s is 28,275 bytes, both far below the limit.

Step 4: Upload It

The file must sit in the root directory of the domain so it is reachable directly at /robots.txt. A robots.txt at /blog/robots.txt is ignored by every crawler. For static sites (Astro, Hugo, Next.js export) put it in the public/ folder; on classic servers upload it to the web root via FTP/SFTP.

If you run your own server with Nginx or Caddy, you can also serve it straight from the server configuration without a file. In Nginx, for example:

location = /robots.txt {
    default_type text/plain;
    return 200 "User-agent: *\nDisallow: /tmp/\n\nSitemap: https://example.com/sitemap.xml\n";
}

For a staging environment this is handy, because the block cannot accidentally travel with the code to the live site.

Step 5: Test It

Test the file before relying on it:

• Google Search Console → Settings → robots.txt report. It shows which version Google fetched last, when, and whether there were errors. Google retired the old interactive robots.txt tester at the end of 2023.
• Bing Webmaster Tools still has a robots.txt tester for checking individual URLs. More on Bing in our article on Bing SEO.
• With a parser on the command line if you want to be really sure. Our test above shows why it pays to ask more than one.

Changes then take a while to take effect. Google generally caches robots.txt for up to 24 hours. You can request a refetch in Search Console.

Who Reads Your robots.txt? Our Log File Analysis

We analyzed every request to getmind.io from October 9 to 19, 2026: 101,447 requests, 1,293 of them for /robots.txt. The most frequent fetchers by user agent:

ClaudeBot fetches the file about twelve times a day (115 fetches in 9.4 days), Googlebot around twenty. GPTBot, on the other hand, fetched it only three times in ten days while requesting 1,673 other URLs. That is not a violation: a crawler may cache the file, and Google explicitly says it does so for up to 24 hours.

Important caveat: these numbers are based on the user agent, which is the client’s own claim. In our AI crawler log analysis we showed that 93% of requests claiming to be AI bots were fake. So the table shows who claims to read the file, not who actually does. For the core point, that legitimate crawlers fetch robots.txt regularly, it is good enough, because the order of magnitude matches the verified numbers from that earlier analysis.

AI Crawlers in robots.txt: GPTBot, ClaudeBot and Friends

For many site owners this is now the main reason to create a robots.txt. The big AI companies each run several crawlers with different jobs, and that distinction decides what you should block:

The split between training and search is the key. Blocking GPTBot stops OpenAI from using your content for future models. Your pages can still appear in ChatGPT search results as long as OAI-SearchBot is allowed. If you block every bot with “GPT” or “Claude” in its name, you may disappear from AI search as well, which is exactly where a new traffic channel is emerging right now. How this affects visibility in Google is covered in Google AI SEO.

A typical “no training, yes to search” configuration:

# No AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including search crawlers
User-agent: *
Disallow: /search

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row form one shared group, which saves repetition. Note that the AI crawlers in the first group follow only that group. The Disallow: /search rule from the * group does not apply to them, which does not matter here because they are blocked from everything anyway.

How Do Major Websites Handle It?

On October 19, 2026 we fetched and counted the robots.txt files of 18 well-known websites, a mix of German and international sites. Result: 12 of the 16 sites that served a real file mention GPTBot explicitly, including nytimes.com, github.com, amazon.de, spiegel.de, zeit.de and faz.net. Spiegel and Zeit, two major German news outlets, block GPTBot completely with Disallow: /.

The outliers are the interesting part:

• reddit.com needs no AI list, because its file simply blocks every crawler with User-agent: * and Disallow: /, and governs access through contracts instead.
• wikipedia.org has the longest file in the sample at 28,275 bytes but does not mention GPTBot. Wikipedia mostly blocks crawlers that caused excessive load.
• heise.de, a large German tech publisher, returned status 410 Gone for /robots.txt, even with a browser or Googlebot user agent. Under Google’s rules a 4xx means: there is no robots.txt, everything is allowed. Whether that is intentional or bot protection locked out our server, we cannot tell from outside.
• stackoverflow.com replied with 418 I’m a teapot, a joke status code that bot protection systems like to hand to unwanted clients. For a rule-following crawler that formally also means “everything allowed”.

The sample is small and covers a single day. The trend is still clear: among news sites and big shops, blocking AI training has become standard. For a company blog like ours the trade-off is different. We want AI assistants to know and cite our articles, so we do not block any AI crawler.

robots.txt Is Not a Security Tool, and Our Logs Prove It

Read this section before you ever put a secret path in your robots.txt.

robots.txt is public. Anyone can read it, not just search engines. Writing Disallow: /secret-admin/ publishes a list of the places you consider worth protecting.

We checked what this means in practice. Our robots.txt contained Disallow: /tmp/. During the ten days we analyzed there were 148 requests for exactly /tmp/.env, a file in which web applications typically store passwords and API keys. The requests came from 120 different IP addresses. 118 of the 148 requests (80%) came from IPs that had fetched our robots.txt beforehand, half of them within about five seconds.

For comparison: for the generic scans of /.env in the root directory that every website on the internet receives, the share was 70%; for /backend/.env, /api/.env and /app/.env it was 58 to 60% each. So our blocked folder was requested disproportionately often by clients that had read robots.txt first. That is an indication, not proof, since some scanners probe /tmp/ anyway. But it fits the well-known pattern: automated scanners read robots.txt, take every Disallow line as a starting point and append typical file names.

One pattern in detail: many of these requests carried a user agent that looked like an AI crawler, including CCBot. A real crawler that obeys robots.txt would never request /tmp/ at all. Whoever does is very likely not who they claim to be.

Our /tmp/ folder was never dangerous, by the way: all 148 requests got a 404, because there is no .env there. What that means for you:

• Confidential content belongs behind authentication, not in robots.txt. HTTP basic auth, a login or an IP restriction in the web server.
• Files like .env do not belong in the web root. Period.
• Do not list unknown admin paths in robots.txt. If you want to hide them, link them nowhere and protect them with a password. For pages that should not appear in Google, noindex in the HTML head or the X-Robots-Tag header is the right tool.
• Only technology helps against rude bots: rate limits, a firewall like UFW and automatic bans with Fail2ban.

Templates: robots.txt for Typical Websites

Minimal: Most Websites and Blogs

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Allows everything and names the sitemap. For most websites this is the best choice, and it is unambiguous for every parser.

WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

admin-ajax.php must stay reachable because many themes and plugins load content through it. The Allow line is longer than the Disallow line, so longest-match and first-match parsers read it the same way as long as it comes right after the Disallow. For more on WordPress SEO, see our comparison of WordPress SEO plugins.

Online Shop

User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /cart
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?filter=
Disallow: /search

Sitemap: https://example.com/sitemap.xml

The wildcard rules keep crawlers out of sort and filter variants. Before adding them, check in Search Console whether those URLs are actually being crawled. If your shop software already marks them with canonical and noindex, a robots.txt block is even counterproductive: Google can no longer read the noindex. For Shopware we covered the details in our article on Shopware SEO.

Staging and Test Environments

User-agent: *
Disallow: /

Blocks everything. A staging site should always have a password on top. And make sure this file never gets copied to the live site. That is one of the classic mistakes that make a website vanish from Google overnight.

The Most Common Mistakes When Creating a robots.txt

1. Disallow: / on the live site. Usually carried over from staging. Result: Google stops crawling, rankings decay over the following weeks.
2. Allow: / as the first rule. Our own mistake. Harmless for Google, but for first-match parsers it cancels every block that follows.
3. Blocking CSS and JS. Google then does not see the page the way users see it.
4. Trying to remove pages from the index with Disallow. Does not work. On the contrary: a blocked page cannot show its noindex and is more likely to stay indexed.
5. Forgetting group logic. A dedicated Googlebot group completely replaces the * group for Google; all general blocks must be repeated there.
6. Relative sitemap URL. Sitemap: /sitemap.xml is invalid; it has to be the full address.
7. Listing secret paths. See above: robots.txt is the first file scanners read.
8. Wrong encoding or file name. Robots.txt or robots.TXT is not found on many servers. UTF-8 with a BOM can make the first line unusable for some parsers.
9. Persistent server errors. A 5xx on /robots.txt can, in the worst case, make Google stop crawling altogether.

This short tutorial from How To Simplify (May 2026) shows where to find the robots.txt report in Google Search Console, which is the step we recommend after every change.

robots.txt vs. Meta Robots vs. X-Robots-Tag: Which Does What?

These three tools are constantly confused. The short version:

Order is what matters: a crawler has to be allowed to fetch a page to see its noindex. Blocking a page in robots.txt and adding noindex achieves the opposite: Google never sees the noindex. To reliably remove a page from Google, keep it crawlable and set noindex. Only once it has dropped out of the index can you additionally block it if you want to save crawl budget.

Conclusion: Create a robots.txt, but Keep It Short

A good robots.txt is short, unambiguous and contains nothing secret. For most websites a handful of lines is enough: a sitemap line, perhaps a few blocks for search and filter pages, and a deliberate decision about AI crawlers.

Our key takeaways from our own measurements:

• Write rules every parser understands the same way. Python’s standard parser read our five-line file backwards because Allow: / came first. Deleting one line fixed it.
• Legitimate crawlers follow it and check back often. ClaudeBot fetched our file around twelve times a day, Googlebot about twenty.
• robots.txt protects nothing. 80% of requests for a .env file in our blocked folder came from IPs that had read robots.txt first.
• Separate training and search for AI crawlers. Block everything and you also lose visibility in ChatGPT, Claude and Perplexity.

To learn which of these crawlers are real and which only pretend, read our AI crawler log analysis.

FAQ: How to Create a robots.txt

How do I create a robots.txt file?

Open a plain text editor, write your rules (at least User-agent: * plus a Disallow or Sitemap line), save the file as robots.txt in UTF-8 and upload it to the root directory of your domain. It must be reachable at https://your-domain.com/robots.txt with status 200. Then check it in the robots.txt report in Google Search Console.

Does every website need a robots.txt?

No. If the file is missing and the server returns a 404, all crawlers assume everything is allowed. It becomes useful as soon as you want to exclude areas, control AI training or announce your sitemap. A minimal version with a sitemap line never hurts.

Where does the robots.txt file go?

Always in the root directory of the host, directly at /robots.txt. Every subdomain and protocol needs its own file: shop.example.com is not covered by the robots.txt on www.example.com.

Can I remove a page from Google with robots.txt?

No, it is the wrong tool for that. robots.txt only prevents crawling. A blocked page can stay in the index if other pages link to it. To remove it, use noindex and keep the page crawlable until Google has dropped it.

How do I block AI crawlers like GPTBot and ClaudeBot?

Create a dedicated group with User-agent: GPTBot, User-agent: ClaudeBot (and optionally Google-Extended, CCBot, Applebot-Extended) followed by Disallow: /. If you still want to appear in AI search, keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Those crawlers handle search, not training.

Do all bots obey robots.txt?

No. Reputable search engines and the big AI companies say they follow it, and our logs support that for verified bots. Scanners, scrapers and bots with spoofed user agents ignore it or even use it as a map to interesting paths.

What is the difference between Disallow and noindex?

Disallow in robots.txt forbids fetching a URL. noindex in the HTML or an HTTP header allows fetching but forbids adding the page to the search index. Using both on the same page does not work, because the crawler never gets to see the noindex.

How long does it take for robots.txt changes to take effect?

Google generally caches the file for up to 24 hours, but may use an old version longer during server problems. You can request a refetch in the robots.txt report in Search Console. Other crawlers have their own intervals. In our logs ClaudeBot fetched the file about every two hours, GPTBot three times in ten days.

Should I put Allow: / in my robots.txt?

No. It is redundant, because anything not forbidden is allowed anyway. It can also do harm: parsers that use “first matching rule wins”, including Python’s standard library, then ignore every later Disallow line. That happened to us on getmind.io.

Does Google respect Crawl-delay?

No. Google ignores Crawl-delay entirely. Bing and Yandex respect it. If Googlebot puts too much load on your server, temporarily return status 429 or 503. Google then throttles automatically.

How large can a robots.txt file be?

Google processes at most 500 KiB; anything beyond is ignored. In practice this rarely matters: the longest file in our sample (Wikipedia) was 28 KB. If your file keeps growing, that is usually a sign you should consolidate rules with wildcards.