Crawler

A crawler is an automated program that visits pages on the web, fetches their content and follows links onward to new pages. Search engines use crawlers to discover and refresh the material their index is built from. Googlebot is the best known, and Bing, Apple and a growing set of AI services run their own.

How a crawler works

The program starts from a list of known URLs and a site's sitemap, fetches the pages, reads the content and records every link along the way. New links join the queue, and so it works across the web page by page. Modern crawlers including Googlebot also render JavaScript, so they see a page roughly as a visitor does. Access stays under the site owner's control: robots.txt can welcome crawlers or turn them away, bot by bot.

AI crawlers changed the calculation

Alongside the classic search bots, GPTBot, ClaudeBot and PerplexityBot now crawl at scale, both to train models and to look up fresh sources. Whether to open or block them has become a strategic question: blocking protects the content and also removes the site from the AI answers where a growing share of future customers look.

For visibility in classic search and AI search alike, one rule holds: a crawler that cannot fetch a page cannot surface it. Crawler access is the foundation under all SEO. What to open, what to close and for which bot is the subject of the guide to crawling and indexing.