Crawling

Crawling is the process where a search engine's crawler automatically fetches pages on the web and reads their content. It is the first link in the chain that runs from crawling through indexing to rankings: a page Google has never crawled cannot appear in the search results, however good it is.

Crawlers find pages three ways: through links from pages they already know, through an XML sitemap, and by returning to URLs they have visited before. The crawler fetches the page, renders it and registers content and links, which can then enter the index. Frequency adapts to the site, so a news section that changes daily gets frequent visits while a static page can go weeks between them.

Crawling and indexing are separate steps, and the distinction matters when pages go missing from Google. A crawled page can still be left out of the index if Google reads it as thin, duplicated or blocked by a directive. A crawl is permission to be considered, not a promise of visibility.

AI services now run crawlers of their own. OpenAI uses GPTBot to collect training data and other bots to look up fresh sources for ChatGPT. Perplexity's crawler, according to Perplexity, is used to find and link to sources in its search results, not to train models. A site that is easy to crawl is visible in two places at once, and that is where all SEO work begins, in classic search and in AI search alike. The guide to crawling and indexing follows the process from first fetch to indexed page.