How to Handle Crawling and Indexing During a Site Migration
The two most damaging failures in this entire subject both happen here. Both are one line long, both make a perfect site invisible and neither shows any symptom whatsoever to a person looking at it. If you take one thing from this cluster, take the check in block three.
The Two Failures That Matter Most
A site can launch with an instruction blocking crawlers from reaching it. A site can launch asking not to be listed in results. Either one makes an otherwise flawless migration completely invisible.
The first. Access blocked.
A file at the root of the site tells crawlers which parts they may visit. If it is still carrying the instruction used to keep the unfinished site private, nothing gets visited at all. The site is live, complete and answering every request, while the things that would list it have been asked to stay away.
The second. Indexing refused.
A separate instruction, held in the pages themselves, asks that they not be included in results. Added during building so the incomplete site did not appear anywhere, then shipped with everything else. This one is arguably worse, because crawlers do arrive, read the site and are politely told to forget it.
What they have in common. Three things.
Each is a single line. Each produces no visible symptom of any kind. And each can be ruled out in seconds by somebody who knows to look.
Why they open this page. Nothing else here competes.
Everything below is worth understanding. These two are worth checking today, before you read the rest, if you have launched anything recently.
Why They Happen
Both of these are the natural consequence of doing the build correctly and forgetting one step at the end. That is worth understanding, because a team that thinks this happens to careless people will not check for it.
Why the protections exist. They are the right thing to do.
A site under construction should not be discoverable. Half-finished pages, placeholder content and draft pricing appearing in results is a genuine problem, so competent teams block the staging site deliberately.
What that means at launch. The protection is part of the build.
It was not bolted on separately. It lives inside the thing being built, so when the build is deployed, the protection deploys with it. Nothing malfunctioned. The instruction did exactly what it was told to do.
Why the last step gets missed. Launch day is crowded.
A dozen things happen at once, several people are involved and the visible checks absorb everybody. Removing a one-line instruction that nobody can see the effect of does not compete for attention against a form that is not submitting or a logo that has come out wrong.
The organisational reading. It belongs on a list.
This is not a knowledge problem. Everybody involved already knows it. It is a sequencing problem, which is why it belongs on the launch checklist rather than in somebody's memory.
Check It Within Minutes Of Launch
This is the first post-launch check, before the redirects and before anything visual. It takes moments. The difference between finding it now and finding it later is the difference between an inconvenience and a serious problem.
What makes the timing matter. Discovery begins immediately.
From the moment the site is live, crawlers start arriving and receiving whatever it tells them. Every hour that a wrong instruction stays in place is an hour of that instruction being acted on. Undoing an instruction is slower than issuing one.
Why it is not caught later. Nothing prompts it.
No error appears anywhere. No customer complains. The site loads beautifully. The only eventual symptom is a decline in performance. A decline after a migration is exactly what everybody was told to expect.
That is the cruel part. The warning masks the fault.
The reassurance that a temporary drop is normal is correct and necessary. It also means the one signal this failure produces is the signal everybody has been told to ignore for a few weeks.
Who should do it. Somebody technical, immediately.
Named in advance and available on launch day, per the site migration SEO checklist.
What The Blocking File Does
Every site can carry a small file at its root, conventionally named robots.txt, which tells crawlers which parts of the site they are asked not to visit. It is a request rather than a barrier.
What it controls. Access, not listing.
It governs whether a crawler goes and looks at something. That is a different question from whether an address appears in results. Conflating the two causes most of the confusion in this area, including among developers. Block five is entirely about why the distinction has practical consequences.
What it is legitimately for. Ordinary housekeeping.
Keeping crawlers away from parts of a site that produce endless variations, such as internal search results or filtered listings, so that effort goes to pages that matter. On a large site that is genuinely useful housekeeping rather than a defensive measure.
Why it matters during a migration. One line changes everything.
The same mechanism used to keep a staging site private can be applied to the whole site. Applied to everything and left in place at launch, the entire site is asking not to be visited. That is the first of the two failures. It originates in a file most people never open.
What this page will not show you. Its contents.
We do not publish what goes inside it. Knowing what it does is what lets you ask whoever runs the site a precise question. Editing it is their job.
Blocking Is Not The Same As Removing
Blocking access to a page does not necessarily keep it out of results. The two instruments do different jobs. Using the wrong one can produce the opposite of what was intended.
What the documentation states. Checked August 2026.
The vendor's own guidance says that while content blocked by the file will not be crawled or indexed, a disallowed address might still be found and indexed if it is linked from elsewhere on the web. It adds that the address and publicly available information such as link text may still appear in results.
What it names as the proper method. Three options.
The same guidance names password protection, the do-not-index instruction, else removing the page entirely as the ways to keep an address out of results.
The interaction that catches people. The two cancel each other.
The indexing documentation states that for the do-not-index rule to work, the page must not be blocked by the file and must be reachable by the crawler, because a blocked crawler never sees the rule and the page can still appear. Applying both at once means the second one is never read.
Why this belongs in a migration guide. Both get used at once.
A team wanting a staging site fully hidden reaches for both instruments together, which is intuitive and counterproductive. Understanding that is what makes the launch check meaningful rather than mechanical, because it explains why removing one protection may not be enough on its own.
Sitemaps
A sitemap is a list of a site's addresses, published so that crawlers can find them without relying on links alone. During a migration it does one job particularly well.
The migration purpose. Getting the new addresses found quickly.
After a launch, the new addresses exist and almost nothing links to them from outside yet. A list published directly is the fastest route to them being discovered, rather than waiting for them to be found by following links.
What it does not do. Guarantee anything.
Submitting an address is telling something it exists rather than requiring it to be included. A sitemap is an invitation and it is frequently described as more than that, particularly by anybody selling a service that involves submitting one.
What should be in it. The new addresses.
The site as it now is. Old addresses that now redirect do not belong in the new list, because the list describes the current site rather than its history. That is a separate question from whether the old list should still exist somewhere.
The exception worth knowing. Block seven.
That is a statement about the new sitemap. The old one is a separate question and it should not be deleted immediately.
Do Not Delete The Old Sitemap On Day One
Keeping the old address list available for a period helps the old addresses get revisited, which is how their redirects are discovered. Removing it immediately slows the whole process for no benefit.
The mechanism. Redirects are found by being visited.
A redirect does nothing until something arrives at the old address and receives it. The faster the old addresses are revisited, the faster the moves are understood, which makes anything accelerating that revisiting worth keeping for a while.
Why the old list helps. It is a complete inventory.
It names every old address in one place, including ones nothing links to any more. Those are exactly the addresses least likely to be revisited soon by any other route.
Why it feels wrong. It looks like leaving the past around.
Publishing a list of addresses that no longer serve content seems untidy, which is why it gets deleted as part of the clean-up. The untidiness is the point for a short period.
How long. Until the moves are processed.
Not indefinitely, nor a fixed period either. Once the old addresses have been revisited and their redirects acted on, the list has done its job, per notifying Google of a migration.
Crawl Rate And Large Sites
How much of a site gets visited, plus how quickly, is a genuine consideration on very large sites and something most readers can safely ignore. Being straightforward about that is more useful than treating it as universal.
Who this affects. Sites with very many addresses.
Large stores, publishers and anything generating addresses automatically. A site of a few dozen or a few hundred pages does not have this problem and never will, which is worth saying plainly because the subject gets discussed as though it applied to everybody.
What the documentation says. Checked August 2026.
The vendor's site move guidance states that for a move to be considered complete, every address on the old and new site has to be visited at least once. It adds that there are no fixed frequencies, with speed depending on the size of the site and the crawling speed possible.
Why that matters here. The vendor is declining to give a timescale.
That is worth noticing, because plenty of third parties will give you one. The organisation actually doing the crawling says it depends on your site.
What a migration does to the demand. Doubles the work.
Every old address needs revisiting and every new one needs assessing. That is why the mapping and the sitemaps matter more on a large site than anywhere else. It is also why a large site should expect its settling period to run longer than a small one's.
Staging Sites That Get Indexed
The opposite problem to blocks one and two. An unprotected build environment can be found and listed, creating a complete duplicate of the site at an address nobody intended anybody to see.
How it gets found. Nothing has to be published.
A link in an email, a reference in a shared document, an address mentioned in a public forum by somebody asking for feedback. It does not require anybody to publish anything deliberately, which is why relying on the address being obscure is not a protection.
What the consequence is. A competing copy.
An entire second version of the site, frequently with unfinished content and draft pricing, existing publicly. It competes with the real site and it can persist after launch, because it does not disappear when the finished version arrives.
Why it is worse during a migration. Timing.
The staging site is a near-copy of what is about to launch. Two nearly identical sites appearing at the same moment is the least helpful thing that could happen during a settling period.
How it is prevented. A decision, made early.
Restricting access at the environment level rather than relying on instructions that ask politely. That is a conversation to have with whoever builds, before building starts. It has the useful side effect of removing the need for the instructions that later get shipped by mistake.
What To Check And When
The same small set of questions, asked at four points. The sequence matters more than the contents.
Before launch. On the staging build.
That the environment is genuinely restricted rather than politely asking not to be visited, plus that everybody knows which protections will need removing.
Within the hour. The two failures.
Whether access is blocked and whether the site is asking not to be listed. Nothing else on any list outranks this.
Within the day. The supporting pieces.
That the new sitemap is published and reachable, that the old one is still available, then that redirects are behaving on live addresses.
Within the week. Confirmation.
That new addresses are beginning to be visited and that nothing is reporting access problems. What to watch beyond that is in how to migrate without downtime. The full series is on the website migration guide.
One line.
No symptom.
Whole site gone.
The two failures checked within the hour rather than discovered in week three, the staging environment restricted properly rather than asked politely, the old address list kept until the redirects have been found, crawl rate treated as a large site question because that is what it is, with nothing published that anybody could edit unsupported.
What a migration engagement covers:
A migration is a one-off project rather than a monthly service, so it is scoped and quoted against the size of the site and how much is changing at once.
Every guide.
One practice.
What a migration is, the types, when to do it, planning, the checklist, auditing, mapping, redirects, internal links, tags, downtime, crawling, structured data, performance, notifying, monitoring, recovery and the mistakes.
