robots.txt
A request to crawlers about which areas to skipA text file in the root directory of a website that tells search engines which areas not to fetch — a request, not a lock.
- Crawler reads robots.txt first
- Blocked paths are not fetched
- Still possible: listing without a fetch
- noindex reliably controls inclusion
What does robots.txt mean?
robots.txt is one of the oldest conventions on the internet. It goes back to Dutch developer Martijn Koster, whose server was overloaded by a crawler in 1994. The informal proposal that resulted, the Robots Exclusion Protocol, became a de facto standard over decades that reputable search engines observe — it was only formally standardized by the IETF in September 2022, as RFC 9309.
Technically the file is about as simple as it gets. It sits at the address yourdomain.com/robots.txt and consists of lines like User-agent, followed by Disallow or Allow and a path. That lets you tell a specific crawler, or all of them at once, which directories to avoid. The file can also reference the sitemap, the address directory of the website, so the two files work together instead of contradicting each other.
The most expensive misunderstanding here is always the same one: robots.txt does not reliably prevent indexing. A blocked page can still show up in search results — without a description, just the bare address — if other websites link to it. The search engine has not fetched the page, but has inferred its existence from external signals and listed it anyway, with a note that no description is available.
Anyone who wants to reliably keep a page out of search needs the opposite of a block: a noindex directive in the page's own header, or an access restriction. That requires the page to be fetchable in the first place — a page blocked via robots.txt can't even have its noindex directive read. The two tools partly cancel each other out in effect, rather than adding up.
In practice we mostly see two mistakes. The first: a Disallow: / line left over from the build phase that gets forgotten and, when the site moves to the live domain, blocks the entire website — the pages load normally in a browser, only a search engine respects the rule and stays out. The second: blocked directories containing CSS or script files, which stops Google from rendering the page correctly enough to evaluate it.
That's why we check the file live rather than in the source folder — we fetch it at the address actually being served, because a local copy can diverge after a move or a server change. What matters is the boundary with its two neighboring files: robots.txt governs access, the sitemap lists the addresses, and whether a fetched page actually gets included is decided only by indexing.
Check the numbers
robots.txt was only formally standardized in September 2022 — 28 years after its first use in 1994.
Almost three decades lie between first use and official standardization, during which search engines voluntarily observed an informal convention. That shows how much the file's effect rests on cooperation rather than technical enforcement — a crawler that doesn't want to comply is not stopped by robots.txt. For small and mid-sized businesses that means: the file is a note to well-behaved systems, not a firewall.
Source: IETF, RFC 9309 "Robots Exclusion Protocol," September 2022, and robotstxt.org on its origin in 1994.
Common questions
Does robots.txt stop my page from appearing on Google?
Not reliably. A blocked page can still appear in results with its bare address if other websites link to it. For reliable exclusion, a noindex directive on the page itself is the right tool — and the page has to be fetchable for that to work.
Does every website need a robots.txt?
No, there's no legal requirement. If the file is missing, crawlers assume everything may be fetched. It becomes useful mainly for deliberately excluding specific areas, such as internal search results or test environments.
A term missing? Send it to us
We explain every term calmly and without tech-speak — and tell you honestly what makes sense for you and what doesn't.
