What Is Log File Analysis in SEO?
Everything else in measurement is sampled, estimated or reported by somebody else. A server log is a record of requests that were genuinely made. That difference in kind is the whole argument for the work. It is also why this only matters above a certain size.
The Only Record Of What Actually Happened
A server log records what your server was asked for and what it returned. Not a sample of it, not an estimate of it and not somebody else's summary of it. The requests that were actually made.
What everything else is. Second hand, in one way or another.
Analytics depends on code running in a browser, which fails, gets blocked and misses anything that is not a person. Search reporting is filtered, aggregated and delayed. Third party estimates are models built from panels and inference.
Why none of that is a criticism. Those sources answer different questions.
Each is well suited to what it was built for. The point is that not one of them can tell you what your server was asked for, because none of them is looking at your server.
What the record contains. The unglamorous facts.
What was requested, when, by what, then what came back. That is a narrow set of facts and it is the only place several important questions can be answered at all.
Why this is not routinely done. Effort and access.
It requires getting hold of the files, which is block seven, plus knowing what to look for, which is most of this page. Neither is difficult. Both are more work than opening a dashboard.
What It Answers That Nothing Else Can
Four things. The important property of all four is that no other source answers them. This is not a better view of something you already have.
Which pages were actually fetched, plus how often. Not which were indexed.
Being fetched precedes everything else. Knowing what receives attention and what does not is the foundation of the other three answers.
Which were never fetched at all. The one with no substitute.
Absence cannot be inferred from a report that only describes what exists. This has its own block below because it is the strongest argument for the work.
What response each request received. As it happened.
Errors, redirects and successes as they were actually served, rather than as they behave when somebody checks a page by hand later. Intermittent faults are invisible to spot checking and obvious in a log.
How attention is distributed across sections. The allocation picture.
Which parts of a site receive most attention and which receive almost none. On a large site that distribution is frequently the opposite of what anybody intended. No other source reveals it.
The Question It Exists To Answer
On a large site the problem is frequently that valuable pages are rarely fetched while worthless ones are fetched constantly. That is invisible without logs and it is the situation this work exists to find.
Why it happens without anybody deciding. Structure produces it.
Attention follows links. A site's internal structure was not designed with attention in mind. Sections nobody thinks about can be extensively linked, while the pages that earn money sit several steps from anything.
What it looks like when you find it. Consistently surprising.
Filtered listings, old campaign material, printer versions or a section retired years ago receiving constant attention, while product or service pages are visited rarely. Nobody built the site that way on purpose.
Why the totals hide it. Averaging.
An overall figure for how much attention a site receives can look entirely healthy while being distributed almost entirely to things that do not matter. The total is not the finding. The distribution is.
What you do about it. Not covered here.
Logs tell you the allocation is wrong. Changing it is a structural exercise covered in what is crawl optimisation at scale. This page owns only the diagnosis.
Pages That Were Never Fetched
A page nothing has ever requested cannot appear anywhere. It is not performing badly. It has not been assessed at all. No other source will tell you it exists in that state.
Why no other report shows this. They describe what happened.
Analytics shows pages people visited. Search reporting shows pages that appeared. Both describe pages that got somewhere. A page nothing ever asked for produces no data anywhere, so it is absent from every report rather than showing as a problem in one.
How it goes unnoticed. It looks like underperformance.
A business sees a section producing nothing and concludes the content is weak or the market is competitive. Both diagnoses lead to rewriting pages that were never read by anything.
How common it is. More than anybody expects.
On large sites it is routine rather than exceptional. Sections reachable only through filters, pages linked from one place that itself is rarely visited, else material behind navigation that gets assembled rather than delivered, as covered in what is JavaScript SEO.
Why it changes what you spend. Different problem, different money.
Rewriting unreachable content is expensive and achieves nothing. Making it reachable is usually cheap and achieves everything. You cannot tell which situation you are in without looking.
Verifying What Claims To Be A Crawler
A great deal of traffic claiming to be a search engine is not. Any analysis done without accounting for that is distorted, sometimes severely. The distortion runs in the direction of flattery.
Why the claim is not evidence. It is self-declared.
A request states what it is. Nothing about that statement is checked by the act of making it, so anything can say it is anything. A great many things do.
What is actually out there. A mixture.
Monitoring services, third party analysis tools, scrapers collecting content, security scanners and things declaring themselves as major search engines for no better reason than that it gets them treated politely.
What the distortion does. Makes the picture look better.
Counting unverified requests as search engine attention inflates every figure you would draw from a log. A business concludes its pages receive plenty of attention when the genuine attention is a fraction of it.
Why we publish no method. Deliberate.
Verification is a solved problem for anybody doing this work properly and we are not going to set out the mechanism. Publishing how identity is confirmed also publishes what somebody would need to defeat it.
What to ask instead. One question of your supplier.
Ask whether their analysis verified the requests or accepted what they claimed. Anybody doing this properly answers immediately.
It Is Not For Small Sites
A site of thirty pages has nothing to learn from this. Everything on it gets fetched, allocation is not a meaningful concept at that size and the analysis would produce a document confirming that nothing is wrong.
Why size decides it. Allocation needs something to allocate between.
The entire value of this work is establishing where attention goes when there is not enough to go everywhere. On a small site there is enough, so the question does not arise.
Roughly where it starts to matter. When you cannot hold the site in your head.
Not a page count so much as a threshold of comprehension. If somebody can list your sections and know what is in each, you do not need this. If nobody in the business could produce that list, you might.
What else raises it. Generated addresses.
Filters, parameters and pagination create far more addresses than pages, so a site with a few hundred products can behave like a much larger one. That is a stronger signal than the page count.
Why we are saying this. It removes a sale.
Recommending this universally would be selling something unnecessary. Saying plainly that most sites do not need it is what makes the recommendation worth anything when a site genuinely does.
Getting Hold Of Them Is The Hard Part
Access depends entirely on the hosting arrangement. This should be established before the work is proposed rather than after it has been agreed.
Where it is straightforward. Your own infrastructure.
A site on a server you or your developer control means the records exist and somebody can retrieve them. This is the easy case and it is less common than it used to be.
Where it is possible with effort. Managed hosting.
Many managed providers can supply logs on request, sometimes only for a recent period. The retention window matters, because analysis needs enough history to show a pattern rather than a fortnight.
Where it is not available. Some hosted platforms.
Certain platforms do not provide server logs to customers at all. That is a legitimate product decision and it means this work cannot be done, which is worth knowing before anybody quotes for it.
The intermediary complication. Anything sitting in front.
Where a delivery network or caching layer sits in front of your site, some requests may never reach the origin server, so the origin's records are incomplete. The useful records may be the intermediary's rather than yours.
What to establish first. Three things.
Whether logs exist, how far back they go and who can retrieve them. A supplier proposing this work without asking is proposing something they have not checked is possible.
What It Pairs With
Logs tell you what was fetched. Other sources tell you what happened afterwards. The combination is where the value is. Either alone answers half a question.
Fetched against appearing. The most useful pairing.
A page fetched regularly and appearing nowhere has a different problem from a page never fetched at all. Search reporting supplies the second half of that. Our Search Console material covers what it shows.
Fetched against published. The completeness check.
Comparing what you believe you publish against what has actually been requested finds the gap. The gap is where the work is. That comparison needs both sides.
Fetched against visited. The commercial layer.
Pages receiving attention but no human visits, plus pages receiving visits despite little attention. Both are informative and neither is visible from one source.
Why the pairing matters more than the log. A log alone answers little.
On its own it is a large file of requests. It becomes useful when set against what you expected, which means somebody has to know what was expected before they open it.
What that implies about the work. It is not a report.
Anybody delivering a log analysis as a standalone document has produced description rather than diagnosis. The full series is on the advanced SEO guide.
Which of your
pages has nothing
ever asked for?
If a section produces nothing, the first question is whether anything has ever requested it, because rewriting unreachable content is expensive and achieves nothing. We check whether logs are even available before proposing this. We will also tell you plainly if your site is not large enough to need it.
What the national tier covers:
The national tier is £1,550 a month. It suits businesses competing beyond one town. We will say so if a lower tier would serve you better.
Every guide.
One practice.
Semantic search, machine learning, topical authority, programmatic production, rendering, log files, crawl allocation, information architecture, content pruning and how strategy changes with scale.
