AI crawlers, robots.txt and content controls: what each directive does

Executive brief

Key takeaways

• Map the purpose and owner of every user-agent.
• Do not use robots.txt as a substitute for authentication or noindex.
• Blocking crawling can prevent a bot from seeing a directive in the HTML.
• Test rules and monitor logs before expanding a block.

“Block AI” is not one technical decision. A system may crawl pages for search, index content, generate snippets, retrieve sources for an answer, train models or perform an action for a user. Each purpose can use a different crawler or control.

Before editing robots.txt, write the policy in plain language: which content, which agent, which purpose and which consequence the business accepts. Then translate that decision into technical directives.

Five uses that should not be confused

Crawling

Crawling is an automated request for a URL. The robots.txt file at the site root tells specified user-agents which paths they may request. Legitimate crawlers usually respect the protocol, but it is not a security mechanism.

Indexing

Indexing is inclusion in a search index. A crawler must access a page to read a noindex directive in its HTML. Blocking the URL in robots.txt and expecting the bot to see noindex is contradictory.

Snippet display

Directives such as nosnippet, max-snippet and data-nosnippet control how much content can appear in supported search experiences. They are not the same as removing the page from an index.

Model training or improvement

Some vendors publish specific tokens that allow sites to control use in model training or product improvement. Scope depends on each vendor's current documentation. Do not assume that blocking a search crawler controls training, or the opposite.

Retrieval for answers and agents

AI products may consult the web to ground an answer or execute a task. Access may happen through a search crawler, a dedicated bot, an automated browser or a user-initiated request. Policy should account for discovery and customer utility.

What robots.txt does

A basic file can allow the entire site:

User-agent: *
Disallow:

Or block one area for one agent:

User-agent: ExampleBot
Disallow: /experimental-area/

The file controls paths; it does not understand editorial intent. Broad rules can affect product pages, documentation and content that should remain discoverable.

Google's robots.txt guide also explains that it is not the mechanism for keeping pages out of the index or protecting sensitive information.

What noindex does

Use a meta tag when a page can be accessed but should not appear in an index:

<meta name="robots" content="noindex">

For non-HTML resources, an X-Robots-Tag header can provide an equivalent instruction:

X-Robots-Tag: noindex

The crawler must receive the response to process the directive. If content is private, noindex is still insufficient: anyone with the URL can access it.

Snippet controls and partial content

When the intent is to limit the displayed excerpt without removing the page from search, preview controls are more specific.

• nosnippet prevents a snippet in supported contexts.
• max-snippet limits text length.
• data-nosnippet excludes specific HTML sections from supported snippets.

These controls can affect discovery and clicks. A page without an excerpt may remain eligible but lose useful context. Test by page type rather than applying a domain-wide rule immediately.

Googlebot and Google-Extended are different controls

Google documents Google-Extended as a product token for controlling uses in specified AI systems. The company says it does not affect inclusion or ranking in Google Search. Generative features inside Search use controls associated with Googlebot and Search previews.

This difference illustrates the general rule: read the user-agent documentation before creating a directive. Similar names do not imply the same purpose.

Consult the current list of Google crawlers and product tokens at implementation time. Agents, products and scopes change.

Build an inventory before blocking

Field	Question
User-agent	Which identifier appears in the request?
Owner	Who publishes its documentation?
Declared purpose	Search, training, answer retrieval, agent or another use?
Accessed content	Which directories and page types?
Value	Does access help discovery, customers or partners?
Risk	Is there unwanted use, cost or exposure?
Control	robots.txt, authentication, meta, header or firewall?
Owner inside the company	Who approves and reviews the rule?

Without an inventory, rules accumulate and remain after strategy changes.

Verify identity and behavior

User-agent text can be forged. When a vendor provides verification through reverse DNS or published IP ranges, use it before trusting requests or creating firewall exceptions.

Analyze logs to understand volume, requested pages, response status, infrastructure impact and behavior after a rule. Logs show requests; they do not prove how content is used later.

Avoid broad blocks without a test

A Disallow: / rule looks simple but can have unexpected commercial effects. Documentation, support and catalog pages may stop serving as useful sources for customers who use assistants.

Roll out progressively:

1. define the policy;
2. select a low-risk directory or content type;
3. validate syntax;
4. monitor logs and discovery surfaces;
5. confirm that essential crawlers still reach required resources;
6. expand only when the result matches the decision.

Policies by content type

• Public acquisition content: may benefit from broad discovery.
• Public documentation: can improve support and adoption when retrieved correctly.
• Licensed content: may require specific contractual limits.
• Authenticated areas: must be protected at the server, regardless of bots.
• Staging environments: need authentication, not only Disallow.
• Internal results and infinite filters: may need crawl management for efficiency.

Audit the controls

Record the current robots.txt, robots meta tags, headers and preview controls. Compare source and rendered output. Check whether a CDN, WAF or challenge blocks agents beyond what the policy states.

Then connect every rule to a business reason. A directive without an owner or justification should be investigated, not removed automatically.

Crawler governance is continuous. The goal is not to allow everything or block everything, but to align public access, discovery, rights and customer utility with controls that match each purpose.

Direct answers

Frequently asked questions

Does robots.txt keep a URL out of Google?

Not necessarily. The URL may be discovered through links and appear without content. To prevent indexing, use noindex while allowing Google to access the directive, or restrict access at the server.

Does Google-Extended affect Google Search?

Google documentation states that the Google-Extended token does not affect inclusion or ranking in Google Search. It controls specific uses in AI products described by Google.

Does blocking a bot protect private content?

No. robots.txt is a public, voluntary instruction for crawlers. Private content requires authentication and server-side access controls.