Indexing

When someone searches, a search engine does not search the internet — it searches its own directory. That directory is called the index, and the process by which a page gets into it is called indexing. As long as your page is not in the index, it simply does not exist as far as search is concerned — no matter how good it looks and how much it cost.

The route there has three steps, and they are worth keeping apart. First the search engine has to learn that the address exists at all: through a link, through your sitemap or through a submission. Then a retrieval program — a crawler, in the jargon — fetches the page and renders it as a browser would. And only after that is it decided whether the content goes into the directory.

It is exactly this last decision that most people overlook. Fetched does not mean taken in. A search engine may leave out any page it considers superfluous — because the same text already sits under a different address, because the page has almost no content of its own, or because nothing within the website links to it. There is no entitlement to be included, and nobody can promise it.

When a page is not in the index, the causes are usually mundane and homemade. A switch called noindex is set and explicitly tells the search engine to leave the page out — often a leftover from the build phase, when the website was not yet meant to be public. Or a file called robots.txt blocks off a whole section. Or a canonical tag points to a different address and thereby declares the page a copy of it.

You can check this yourself without buying any tools. In the Google Search Console — the free access that every site owner can set up for their own domain — there is a coverage report and a single-address lookup. It states in plain words whether a page was taken in, and if not, for what reason. A rough first look is also possible through search itself, by typing the command site: followed by the domain.

What makes inclusion more likely is unspectacular: every page should be linked from at least one other page of your website, carry content that exists in that form only there, be reachable under one single fixed address and appear in an up-to-date sitemap. Pages that can only be reached through a dropdown menu or a form are regularly overlooked. And pages that end in a 404 error drop out again over time.

One point that costs real money in practice: after a rebuild, a page can fall out of the index even though it looks flawless in the browser. The reason then lies almost always in a setting nobody checked, because the page was visible after all. That is why every move should be followed by a check that asks not only "does the page load", but also "is it allowed into the index".

We check this by fetching every address from the sitemap individually and logging the result, rather than taking samples. At the last complete check of our own website that was 287 addresses, all reachable. What we expressly do not promise is the inclusion itself: we can create the conditions and document the state, but the decision is the search engine's.