Search engines work in three stages. They find pages by crawling, store and analyse them in an index, and then choose which indexed pages to show for each query. A page has to pass the first two stages before it can compete in the third, and passing them is never guaranteed.
This article follows Google’s own description of the process, because Google is the search engine most Australian businesses care about. Other engines work on the same broad principles.
Stage 1: crawling
Crawling is discovery and download. Google uses automated programs to fetch the text, images and video on a page. It finds new addresses mainly by following links from pages it already knows, and by reading sitemaps that site owners submit.
Three practical consequences follow:
- A page that nothing links to, and that is not in a sitemap, may never be found.
- Google decides how often to crawl and how many pages to fetch from a site. It slows down if a site returns server errors such as HTTP 500.
- Pages behind a login cannot be fetched, and site owners can block crawling with a robots.txt file.
Googlebot also renders pages using a recent version of Chrome and runs their JavaScript, so content that loads through scripts can usually still be seen.
What robots.txt does not do
Google’s documentation is explicit that robots.txt is not a mechanism for keeping a page out of Google. A blocked page can still appear in results, usually without a description, if other sites link to it. To keep a page out of results, use a noindex rule or password protection.
Stage 2: indexing
After crawling, Google analyses the page: its text, its title and image alt text, and other key tags and attributes. It stores what it learns in the index. It also groups duplicate or near-duplicate pages and picks one canonical version to represent the group.
Indexing is where many pages quietly fall out. Google states that not every page it processes will be indexed. Common reasons include low-quality content, robots meta rules that block indexing, and site designs that make indexing difficult.
Stage 3: serving and ranking
When someone searches, Google looks through the index and returns the pages it judges most relevant and useful. Relevance draws on hundreds of factors, including the searcher’s location, language and device. That is why two people can see different results for the same words, and why nobody can promise a particular position.
What you control is the quality and clarity of the page, and whether it matches what the searcher wanted. The search intent article covers that second part in detail.
A quick self-check for your own site
| Question | How to check | If the answer is no |
|---|---|---|
| Is the site indexed at all? | Search site:yourdomain.com.au in Google |
Check technical requirements, then wait a few weeks |
| Can crawlers reach key pages? | Link to them from the home page or main navigation | Add internal links |
| Is a sitemap needed? | Google says small sites (roughly 500 pages or fewer) may not need one | Many CMS platforms generate one automatically |
| Is anything blocked by mistake? | Review robots.txt and noindex rules | Remove the accidental block |
Google’s sitemap guidance adds that a sitemap helps discovery but does not guarantee crawling or indexing. Its SEO starter guide notes that changes can take hours to months to show, so judging results after a few days is premature.
What this means for decisions
- Make pages reachable. Every page you care about should be linked from somewhere sensible on your own site.
- Write for people. Google says helpful, original, well-organised content probably has the biggest influence on search presence.
- Plan migrations carefully. A redesign that changes URLs can break discovery and indexing. See how to plan a redesign without losing SEO value.
- Learn the words your audience uses. A free keyword research method is a sensible next step.
Limits of this explanation
This is a simplified model. Google does not publish its ranking systems in full, and the individual factors change over time. Treat anyone who claims to know exactly how to rank first with caution.
Where Radiance Impact fits
Radiance Impact’s Growth and Marketing division works on search visibility, SEO and strategy, and the Technology division builds business websites with crawlability in mind. If you would like a second opinion on how your site is found, see Growth or get in touch.
Sources
- How Google Search works, Google Search Central
- SEO starter guide, Google Search Central
- Introduction to robots.txt, Google Search Central
- Sitemaps overview, Google Search Central
