How Google Search Works

Google returns results in about half a second from an index of hundreds of billions of pages. Almost none of that work happens when you press enter - the hard parts ran days or weeks earlier.

Understanding the pipeline changes how you think about SEO, because it tells you when each of your decisions gets evaluated. A robots.txt mistake fails at stage one. A JavaScript rendering problem fails at stage two. Thin content fails at stage three. They are different failures with different symptoms, and treating them alike is why so much SEO advice is applied to the wrong problem.

Crawling: finding pages

Googlebot is a program that fetches pages and follows their links.

How it decides what to fetch

It works from a queue built from links it has already seen, sitemaps you submit, and URLs it has crawled before. Every site gets a rough crawl budget - how much attention Google spends - based on how important the site appears and how well the server responds.

For a personal site this is almost never a constraint. It matters at the scale of hundreds of thousands of URLs, where wasting the budget on parameterised duplicates means real pages go unvisited.

Two mechanics worth knowing. Googlebot respects robots.txt before fetching, so a disallowed path is never requested - and that is why it cannot see a noindex on a blocked page. And it re-crawls at a frequency it infers from how often a page changes; a post that never changes gets visited less over time.

Where crawling fails

Almost always one of:

robots.txt blocking too much, often copied from staging.
No links anywhere. Orphaned pages are found by sitemap or not at all.
Server errors or timeouts. Persistent 5xx responses cause Google to back off and crawl less.
Infinite spaces - calendars, filters, and search pages generating endless URLs that consume budget.

Indexing: understanding and storing

Fetching is not keeping. Google parses the page, decides what it is about, and decides whether it is worth storing.

The rendering step in the middle

This is the part that surprises developers. The pipeline is not fetch-then-index; it is closer to fetch, index the HTML, queue for rendering, then index again with whatever JavaScript produced.

That second pass is real but slower and resource-limited. Content present in the HTML is understood on the first pass, reliably. Content that only exists after JS execution waits in a queue.

For a static site or a server-rendered one, this is a non-issue. For a client-rendered SPA, it is the whole ballgame.

Canonicalisation: picking one URL

Google frequently finds the same content at several URLs and must pick one to index. It considers your canonical tag, internal links, redirects, sitemap entries, and HTTPS status - and your canonical is a hint, not a command.

This is why Search Console distinguishes "user-declared canonical" from "Google-selected canonical". When they disagree, Google concluded another URL was the better representative - usually because your internal links point somewhere else.

Why pages get dropped

Crawled but not indexed is common and usually means one of:

Thin content - not enough substance to be worth storing.
Duplicate - materially the same as a page already indexed.
Low value - tag archives, empty category pages, near-identical listings.
Explicit noindex - sometimes intentional, sometimes a template accident.

Search Console reports each of these by name under Page Indexing, which makes diagnosis mechanical rather than speculative.

Ranking: choosing an order

Now the query arrives, and the index has to be sorted in milliseconds.

What happens when you press enter

Roughly: the query is interpreted - spelling, synonyms, and intent, since "apple pie recipe" and "apple stock" want different things. Candidate pages are retrieved from the index. Those candidates are scored by hundreds of signals. Results are assembled with whatever features fit - snippets, images, video, People Also Ask, AI summaries.

The retrieval-then-scoring split is why relevance is a gate, not a slider. A page that is not a plausible candidate for the query is never scored at all, no matter how authoritative the site is.

Signals that are public and stable

Google does not publish the algorithm, but a durable set is confirmed or overwhelmingly evidenced:

Relevance - does the content match the query and its intent?
Quality and depth - is it substantive, original, accurate?
Links - do credible, relevant sites reference it?
Freshness - weighted heavily for news, barely for reference material.
Usability - mobile-friendly, HTTPS, reasonable Core Web Vitals.
Localisation - language and region matching.
Personalisation - modest; location and search history nudge results.

Note what is absent: any lever you can pull mechanically. That is deliberate. The algorithm's goal is to make "be really useful" the cheapest way to rank, and every update since 2011 has pushed further in that direction.

Why results change without you doing anything

Rankings move constantly, and it is rarely about you. Competitors publish or improve pages. Google ships updates - several a day, a handful of large ones a year. Query intent shifts seasonally. Results are tested and personalised.

Which is why chasing a single keyword's daily position is a poor use of attention. Trends over weeks, across many queries, are the signal.

Key takeaways

Three stages, three different failures. robots.txt breaks crawling, JS rendering breaks indexing, thin content breaks ranking.
Rendering is a separate, later pass. HTML content is understood immediately; JS content waits.
Your canonical is a hint. Google picks, and internal links influence the choice more than the tag does.
Crawled but not indexed usually means thin, duplicate, or low value - and Search Console names which.
Retrieval precedes scoring, so relevance is a gate: authority cannot rescue a page that is not a candidate.
Rankings move without you. Read trends over weeks, not daily positions.

FAQ

How long before a new page appears in search?

Days to weeks. A well-linked site with a sitemap and steady publishing gets crawled faster. Requesting indexing in Search Console helps and does not guarantee anything.

Does Google penalise sites?

There are manual actions - human reviews for guideline violations, reported in Search Console. Most traffic drops are not penalties; they are algorithmic re-evaluations, which look identical from outside but have no notification and no appeal.

Is there a sandbox for new sites?

Not officially. New sites do typically take months to rank competitively, which is better explained by having no links and no track record than by a deliberate delay.

How does Google handle duplicate content across sites?

It picks one version as canonical, usually the one it saw first or on the more established domain. Syndicating your writing elsewhere without a canonical pointing home risks the copy outranking your original.

Do AI overviews change any of this?

They change the interface, not the pipeline. Summaries are assembled from indexed pages, so crawlability and credibility still decide whether you are a source. What changes is traffic distribution: simple factual queries increasingly get answered without a click.

Should I care about Bing?

If a meaningful slice of your audience uses it, yes - and the fundamentals transfer. Bing renders less JavaScript, which makes server-rendered HTML matter more, not less. It also powers several AI assistants' web results.

Conclusion

Search looks like magic because the expensive parts happened long before your query. Crawling and indexing run continuously in the background; ranking is the fast lookup at the end.

The practical value of knowing this is diagnostic. When a page underperforms, the question is not "how do I do SEO to it" - it is which stage failed. Not crawled is a robots and linking problem. Crawled but not indexed is a content quality or duplication problem. Indexed but not ranking is a relevance or authority problem.

Three questions, three different fixes. Most wasted SEO effort is the right fix applied to the wrong stage.

Technical SEO Explained goes deeper on stages one and two, On-Page SEO vs Off-Page SEO covers the ranking signals you can influence, and How to Write SEO-Friendly Blog Posts turns it into a writing process.