What Is Robots.txt in SEO?
A small text file with the power to remove an entire site from search results in one line. That combination of high consequence and low complexity is why it deserves attention out of all proportion to its size, plus why one particular mistake matters more than everything else here.
One Line Can Hide Everything
A plain text file, a few lines long, sitting at a predictable address on every site. Written wrongly, it can stop an entire site being fetched. Nothing on the site itself will look any different.
What makes it dangerous. The ratio.
Almost nothing else in technical work combines this much consequence with this little complexity. There is no skill barrier protecting anybody from getting it badly wrong.
Why the damage is invisible. The site still works.
Every page loads, the design is intact and nothing reports a fault. The only symptom is a gradual disappearance from results, which takes long enough that the cause is rarely connected to it.
Who usually writes it. Whoever built the site.
It is created once during a build, frequently by somebody following a template, then nobody looks at it again for years. That is where most of the problems in this guide originate.
What this page will not contain. Any file contents.
No directives, no syntax and no examples. A business owner needs to understand the risk and a developer already has the specification, so publishing the contents serves neither while handing somebody enough to disable their own site.
What it will give you. The judgement.
What the file controls, what it does not, the mistakes that matter and how to check yours in a minute. That is the part nobody is publishing.
What It Actually Does
It controls whether pages are fetched. It does not control whether they appear. That single misunderstanding causes nearly every error described further down this page.
What it governs. Requests.
It tells crawlers which parts of the site they are asked not to request. That is the whole scope of it, operating before anything is read rather than after.
What it does not govern. Presence in results.
Nothing in this file states whether a page should be listed. Those are separate decisions handled by separate mechanisms. Conflating them is where the trouble starts.
Why people assume otherwise. It reads like a wall.
A file whose entire purpose is telling something to stay away sounds like it hides things. It withholds the contents rather than the existence, which is a narrower effect than the name suggests.
What follows from that. The next two blocks.
Because it controls fetching rather than appearing, blocking is not removing. And because it is a published request rather than a barrier, it is not security either.
Where the sequence sits. The found stage.
In the three-stage sequence on crawling and indexing, this file operates entirely at the first stage. Everything it can affect happens before a page is judged.
Blocking Is Not Removing
A blocked page can still appear in results. Using this file to hide something is the wrong instrument. The right one is a different mechanism entirely.
How that happens. Knowledge without content.
A search engine can know a page exists from links pointing at it, without ever fetching it. It may list the address on that basis, with no description because it was never allowed to read the page.
What the result looks like. Worse than either option.
An address listed with no useful description, which is neither hidden nor presented properly. Somebody wanting a page gone has achieved a bare listing instead.
Why the mistake is so common. It is the obvious file.
Somebody asked to keep a page out of search reaches for the file whose name suggests exclusion. The reasoning is sound and the instrument is wrong.
What the right instrument is. A page-level instruction.
Keeping a page out of results is done on the page itself rather than in this file, which requires the page to be fetchable so the instruction can be read. We are not publishing that instruction either.
The trap worth naming. Doing both.
Blocking a page while also instructing that it not be listed prevents the second instruction being read at all. That combination is self-defeating and it appears constantly.
The Disallow Instruction
The instruction that asks a crawler not to fetch a given part of the site. It is a request that well behaved crawlers honour rather than an enforcement mechanism. That distinction has consequences.
What it asks. Please do not fetch this.
An area of the site is named and crawlers are asked to leave it alone. Nothing physically prevents a request, so compliance is a matter of the requester choosing to comply.
Who honours it. The major search engines.
Established crawlers respect it as a matter of course, which is why the mechanism works at all in practice. That is convention rather than enforcement.
Who does not. Everything else.
Scrapers, harvesters and anything built to ignore it will ignore it. Some do not even request the file, so it is not part of their process at all.
Why that matters. It shapes what you can rely on.
You can rely on this to keep tidy areas out of the way of search engines. You cannot rely on it to keep anything away from anybody determined, which is block five.
The breadth problem. Rules apply downwards.
An instruction naming an area applies to everything beneath it. That is how a single broad rule accidentally removes a whole section, which is one of the mistakes in block six.
It Is Not Security
Anything genuinely sensitive is not protected by this file. Worse, listing it here can advertise its existence. Access control is an entirely different discipline.
Why it protects nothing. It is a request.
A file asking politely to be left alone stops nothing that chooses to ignore it. There is no authentication, no barrier and no consequence for non-compliance.
Why it can make things worse. The file is public.
It sits at a predictable address and anybody can read it. A file naming your administrative areas has published a list of the places somebody curious would look first.
The version that genuinely happens. Naming a private area.
Somebody blocks a directory containing something confidential, thinking that hides it. They have instead published its location while providing no protection at all.
What actually protects something. Access control.
Authentication, permissions and keeping sensitive material off a public server. Those are the mechanisms. None of them has anything to do with this file.
The rule worth carrying away. If it matters, do not rely on this.
Anything you would be troubled to see published needs a real barrier. This file is for tidiness rather than for protection.
Common Mistakes
Four failures account for nearly all of it. They are listed by consequence rather than by frequency, because the worst one is not the most common.
A blocking rule shipped from a test site. The catastrophic one.
A file written to keep a development site private, carried across to the live site during launch. The whole site asks not to be fetched. Block seven covers this on its own.
Blocking resources a page needs. The subtle one.
Preventing access to the files a page uses to display properly. The page can be fetched and cannot be understood as visitors see it, which produces judgements based on a broken version.
A broad rule catching more than intended. The accidental one.
An instruction meant for one area applying to everything beneath it, taking a whole section with it. Nobody intended it and nobody noticed, because the pages still load.
The file being absent or unreachable. The mild one.
No file where one is expected, else a server error when it is requested. Usually harmless, since nothing is being blocked, though an error rather than an absence can be treated more cautiously.
What they share. Silence.
Not one of these produces a visible symptom on the site. Every one of them is found by looking deliberately, which is what block eight is about.
The Staging Mistake Is The One That Matters
A blocking instruction written for a test site, shipped to the live one at launch. It removes an entire site from search, it is invisible on the site itself, then it is checked in seconds.
How it happens. Correctly, then carelessly.
A development site is blocked deliberately, which is right. When the site launches, the files are copied across and the blocking instruction comes with them. Nobody wrote it on the live site, so nobody thinks to remove it.
Why it goes unnoticed. Nothing looks wrong.
The new site is live, working and being admired. Traffic decline appears over weeks rather than immediately, by which point attention has moved to other things.
What it costs. Everything, slowly.
A site not being fetched loses presence gradually as stored pages age. On an established site this is the single most expensive technical mistake available. It is also entirely avoidable.
Who discovers it. Usually somebody new.
An agency taking over months later, checking basics as a matter of course. That conversation is uncomfortable for everybody involved in the launch.
How long checking takes. Under a minute.
The file sits at a predictable address and can be read in a browser. Anybody can look, immediately after launch, then confirm it says what the live site needs it to say.
Where else this appears. Migrations.
Our site migrations material opens its crawling page with the same failure, since a launch is a migration and this is the checkable thing that most often goes wrong.
Checking It
Three questions, none of which requires a tool. Knowing what to ask survives interfaces changing, which they do.
What does the file say. Read it.
It sits at a predictable address on your own site and displays in any browser. Whether it contains a broad blocking instruction is visible to anybody in seconds.
Is anything important being refused. The comparison.
Whether the areas named are ones you genuinely want left alone. This is a judgement about your site rather than a technical check, which is why it needs somebody who knows the business.
Is a specific page affected. The targeted check.
Whether a page you care about is caught by any rule, deliberately or through breadth. Testing facilities exist for this and they answer that one question.
What checking will not tell you. Whether the decision was right.
A tool confirms whether a rule applies. It has no view on whether blocking that area was sensible. That is the part that actually matters.
When to check. After every launch and change.
Immediately after a site goes live, plus after any development work that touched configuration. Where the reports live is covered in our Google Search Console material.
Who Should Change It
This file should have one owner and changes should be recorded. Most of the damage in this guide comes from somebody making a change nobody else knew about.
Why one owner. Shared files drift.
Where a developer, an agency and a platform can all alter it, nobody knows what the current state reflects. One named person approving changes prevents most of that.
What recording means. Two sentences.
What changed and why, written somewhere findable. Not a formal process, just enough that the next person can tell whether a rule was deliberate.
Why the record matters more than it sounds. Rules outlive reasons.
A blocking instruction added for a good reason three years ago is indistinguishable from a mistake. Without a note, nobody can tell whether removing it is safe.
Who should not change it unilaterally. Anybody.
Including us. A change to this file can remove a site from search, so it warrants a conversation rather than a quiet edit, however routine it looks.
The minimum viable arrangement. One owner, one note, one check.
Somebody responsible, a record of changes and a look at the file after every launch. That is the whole governance requirement. The sitemap pillar makes a related point about contradicting yourself.
Check it the day
the site goes
live.
A blocking instruction carried over from a test site removes an entire site from search, looks like nothing on the site itself and takes under a minute to find. It is the most expensive technical mistake available and it is entirely avoidable.
What we check at launch:
We publish no directives or file contents. A developer has the specification.
Every guide.
One practice.
How search finds and stores a site, what your own files are telling it, addresses and duplication, status codes, mobile, hosting and what a real audit contains.
