Googlebot

Googlebot is the name of Google's crawler: the program that visits pages across the web, follows links and brings content home to Google's index. Anything that can rank in Google has been visited by Googlebot first, which makes open access for Googlebot a basic condition in technical SEO.

Googlebot crawls with a mobile user agent, because Google indexes the mobile version of the web first. It fetches your HTML, follows internal links and sitemaps, and returns at a frequency that depends on the size, authority and update rate of the site, the so-called crawl budget. It can render JavaScript, but rendering costs resources, so content that only appears in the browser can take longer to reach the index than content sitting directly in the HTML. Access is governed by robots.txt.

Googlebot now shares the web with a growing set of AI crawlers such as OpenAI's GPTBot and PerplexityBot, and Google-Extended controls whether Google may use your content to train its AI models. The consequence differs by crawler: block an AI crawler and you disappear from that service's answers while your Google rankings stay untouched. Block Googlebot and you leave both the search results and AI Overviews, which are built on the same index. The whole chain from crawl to index is unfolded in the guide to crawling and indexing.