Crawling, Indexing & Bot Control

A search engine does three things, and it does them in order: it crawls, it indexes, it ranks. All three steps have to succeed, and none of them can be skipped. If the robot cannot fetch a page it never gets indexed, and a page outside the index cannot rank on anything at all. That is why crawling and indexing are the first things we look at in any technical review, ahead of speed and ahead of structured data. Everything else in SEO depends on these two steps working.

What crawling is

Search engines rely on crawlers, automated scripts that go looking for available material on the web. The crawler starts from a list of sites to visit, and the algorithms decide which pages on that list get fetched, how many of them and how often. From there it moves through your pages by following tags like href and src, hops between internal and external pages, and over time builds a growing map of how the web connects.

A page that cannot be found does not get indexed, and then it does not exist in search. Three things make the robot's job easier:

• A structure with logic in it. A natural hierarchy running from domain to category to subcategory lets the crawler work through the site efficiently. How you build one is covered in taxonomy and architecture.
• Internal links on every page. The robot needs links to move. A page with no incoming internal links, an orphan page, cannot be crawled and therefore cannot be indexed. The method sits in internal link building.
• An XML sitemap. The map of your pages, telling the crawler what exists and what is worth fetching.

What indexing means

Once the page has been found it has to be assessed. During indexing everything on it gets read and organized: text, headings, images, HTML and the concepts the page deals with. That process is how the engine forms a view of what the page is actually about, and therefore which questions it can be relevant for. The information then goes into an enormous index, best pictured as a library with its own section for every word appearing on an indexed page.

Indexing is not a one-off event but a continuing process, which makes it worth keeping the three pillars in order all the time: content, links and technical health.

• Content that answers a real question. With no work put into the writing and the visuals, engines rarely reward the page with visibility. Start from keyword research so you know which questions the page is meant to own.
• Internal links and technical health. Large numbers of 404s, missing meta descriptions and orphan pages make pages look unworthy of visibility. That work sits in site health and redirects.
• Relevant, strong external links. Other sites being willing to point at you is a mark of quality, and it weighs heavily when the engine judges what deserves a place in the index.

Check whether your pages are indexed

New pages, and pages that fall short on those three pillars, frequently do not make it into the index at all. Two quick ways to find out where you stand:

Google Search Console. Select your property and paste the specific URL into the URL inspection tool. You get an immediate answer on whether the page is indexed, when it was last crawled and which version Google has read.

The site: command. Type site:yourdomain.com keyword into the search field and you see which of your pages the engine has indexed on that particular term. The command works in Google and Bing alike, and it is a good way to discover that two of your own pages are competing for the same query.

The Pages report in Search Console is the next step. It holds the reasons pages have not been indexed, and most of them are fixable: discovered but not crawled, crawled but not indexed, blocked by robots.txt, alternate page with proper canonical tag.

The sitemap: the map you draw yourself

An XML sitemap is a file listing the URLs you want the search engines to fetch. It creates no ranking on its own, but it speeds up the discovery of new pages considerably, and on a larger site it is the difference between a new product page being found tomorrow or in three weeks.

Four things decide whether the sitemap does any good:

• Indexable URLs only. A sitemap should contain pages that return 200, are canonical, and carry no noindex. Put redirects, error pages or noindexed pages in the file and you lower the trust in the whole map.
• Automatic generation. The sitemap has to update when the content does. A file somebody exported by hand six months ago points at pages that no longer exist.
• The right size. Maximum 50,000 URLs and 50 MB per file. On a larger site you split it and collect the parts in a sitemap index file.
• Two places to declare it. Put the line Sitemap: https://yourdomain.com/sitemap.xml in robots.txt, and submit the file in Search Console. That also gives you the statistics on how many of the submitted URLs actually got indexed.

Bot control: steering instead of hoping

Robots.txt decides what the crawlers are allowed to fetch. The noindex meta tag decides what is allowed to sit in the index. The two solve different problems, and confusing them is common. A page blocked in robots.txt cannot be read, so the crawler never sees that you put a noindex on it. To get a page out of the index it has to be crawlable and carry noindex, not be blocked.

The next layer is crawl budget. On a large site a significant share of the robot's time goes to pages that do not deserve it: filtered category pages with endless parameter combinations, internal search results, sort variants of the same listing. Clear them out and the pages that matter get fetched more often.

Then there is the layer that is new since AI search became a surface of its own. GPTBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot fetch your pages on their own terms, and your rules apply to them exactly as they do to Googlebot. Here we keep running into the same fault at companies: robots.txt lets them through, but the CDN or the firewall blocks AI crawlers a layer below. You are then invisible in the AI answers however good the content is, and no report will tell you. Check the server logs, not just the configuration file.

Crawl, indexing and bot control in detail is the core of technical SEO, and we deliver it as technical audits. To simply know where you stand today, it starts with a free SEO analysis, where crawl and indexing are the first things we look at.