Robots.txt

Robots.txt is a text file in the root of a domain, typically at example.com/robots.txt, telling crawlers which parts of a site they may visit. It is the first thing a crawler looks up before it starts, and it works as the traffic rules for everything from Googlebot to the bots behind the AI services. The file is a small but load-bearing piece of technical SEO.

The rules sit in simple blocks per user agent, meaning per crawler:

User-agent: *
Disallow: /cart/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The first block applies to every crawler and closes off the shopping cart. The second shuts one bot out entirely, here OpenAI's GPTBot. The sitemap line points crawlers to the XML sitemap.

The file's most current use is control of AI crawlers: GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, which governs how content is used in Google's AI models. The choice is strategic. A block keeps content out of AI answers and training, but it also removes the site from surfaces where more and more people look for suppliers.

Two things the file cannot do. It is no security measure, since it is publicly readable and only well-behaved crawlers respect it. And it does not remove pages from Google's index: a blocked page can still surface if others link to it, and deindexing with noindex is the way out. The interplay between file, index and bots sits in the guide to crawling and indexing.