robots.txt is not access control
This is the misunderstanding worth spending a paragraph on, because acting on the wrong version of it is how private URLs end up public.
Disallow asks a crawler not to fetch a path. It is a request that Google, Bing, DuckDuckGo and other mainstream crawlers honour because they choose to. A scraper does not have to, and an attacker certainly will not. The file itself is served publicly at a fixed, well-known address, so every path you list is readable by anyone who types /robots.txt into a browser. Listing /internal-reports/ to keep it quiet does the opposite: it publishes a directory of the things you did not want found.
Anything that genuinely must not be reached needs a server-side control — authentication, an IP allowlist, an unguessable token, or simply not deploying it. robots.txt belongs to the category of crawl budget management, not the category of security.
Disallowed is not the same as not indexed
These are two different mechanisms and they interact in a way that catches almost everyone.
Disallow governs crawling: whether the bot fetches the page. noindex — either <meta name="robots" content="noindex"> in the head or an X-Robots-Tag: noindex response header — governs indexing: whether the page may appear in results. A URL that is disallowed but linked from somewhere else can still be indexed, because the engine knows the URL exists and can guess at its subject from the anchor text. What appears is a result with the bare URL and no snippet, which is usually more conspicuous than the page would have been.
The trap follows directly. If you disallow a page and put noindex on it, the crawler never fetches the page, so it never sees the noindex, so the tag has no effect. To keep something out of results, allow it to be crawled and serve noindex. Once it has dropped out of the index — which takes a recrawl, not an instant — you can add the disallow if you also want to save the crawl.
| Goal | What to do |
|---|---|
| Keep it out of search results | noindex, crawlable. No disallow. |
| Stop wasted crawling of a large useless section | Disallow, accepting that stray URLs may still be listed |
| Keep it genuinely private | Authentication. robots.txt is irrelevant here. |
| Remove something already indexed, urgently | Removal request in Search Console, then noindex to make it stick |
AI crawlers
GPTBot (OpenAI), CCBot (Common Crawl), ClaudeBot (Anthropic), Google-Extended, Applebot-Extended and Bytespider collect data used in training. Their operators state that they honour robots.txt, and blocking them is a request on the same terms as any other — it excludes you from future collection and does nothing about copies already taken.
AI search bots are separated into their own option because the trade is different. OAI-SearchBot and PerplexityBot fetch pages in order to answer a question and cite the source, which sends traffic back. Blocking those removes you from a surface where you might have been the answer. Google-Extended is worth understanding on its own: it controls use in AI training and Gemini grounding, and blocking it has no effect on ordinary Google Search indexing.
The rules that quietly cost you
Blocking /css/, /js/ or an /assets/ folder was standard advice fifteen years ago and is now actively harmful. Google renders pages, and a page rendered without its stylesheet or its scripts can look broken, unresponsive on mobile, or empty. Let the crawler have the assets.
Disallow: / and Disallow: are opposites — the first blocks everything, the second explicitly blocks nothing — and one slash is the whole difference. Rule groups are separated by blank lines, a crawler obeys the single most specific group naming it and ignores the rest, and a crawler that matches its own name does not also read the * group. The file must sit at the domain root; one in a subdirectory is never fetched. Subdomains are separate: blog.example.com needs its own.
Test the finished file before you trust it. Search Console shows the version Google last fetched along with any parse errors, which is how you find out that the deploy did not actually replace the staging copy.
Questions people ask
Will Disallow remove a page from Google?
Not reliably, and not on its own. Disallow stops the fetch; it does not delete anything already indexed, and a disallowed URL that other sites link to can be listed with no description. The tag that keeps a page out of results is noindex, and the page has to stay crawlable for the engine to see it. Disallowing and adding noindex at the same time is self-defeating.
Can I hide a private directory by disallowing it?
No, and it makes the situation worse. robots.txt is published at a fixed public URL, so the disallow line tells anyone who reads it exactly where to look. The paths remain fetchable. Use authentication or server configuration for anything that matters.
I uploaded it and nothing changed.
Crawlers cache robots.txt, typically for around a day, so a change is not immediate. Confirm the file is actually served at the root of the exact host in question and returns 200 rather than an HTML 404 page. Search Console reports the last fetched version and any errors in it.
Does Crawl-delay reduce load on my server?
Partially. Google ignores the directive entirely and manages its own rate. Bing and Yandex read it. If crawler load is genuinely hurting a server, rate limiting or caching at the server is the control that actually binds, since it does not depend on the client cooperating.
How do I set different rules for one specific bot?
Add a group headed by that bot name and put every rule it needs underneath, including any rules from the * group you still want applied. A crawler follows exactly one group — the most specific match on its name — and does not inherit anything from the others.