Practical Guide: How Website Owners Can Set AI Training Preferences in robots.txt

For website teams, the emerging standard is less about blocking bots and more about publishing a machine-readable intent: which material may help train generative AI systems. Google, Microsoft, OpenAI and Anthropic are backing a draft that lets a site state that choice through a set of rules in robots.txt. The practical value is narrow but useful: teams can express training permission separately from ordinary crawl access, while staying inside a file format that web infrastructure already understands.

What website owners need to decide first

Before editing the file, publishers should separate three questions: should AI systems be allowed to fetch the page; should fetched text become training data; and should the page be retrieved to support an AI answer? The new training preference addresses the second question. It does not automatically answer the first or third, although later sections explain where provider behavior overlaps.

The proposal is tracked as an IETF Internet-Draft on the IETF DataTracker, which means webmasters should treat the wording as a standardization target rather than a finished law-like requirement. Still, because participating vendors have publicly stated their timelines, teams can prepare rules now.

How to write the training preference

Instead of maintaining a vendor-specific checklist, operators can place the shared User-Agent token in robots.txt and pair it with ordinary Disallow and Allow directives. The pattern can cover the whole site, or create exceptions for selected folders such as press kits, changelogs, or public archives.

Publisher goal	robots.txt pattern	Review point
Exclude all site content from AI training	User-Agent: __usage_ai-training; Disallow: /	Confirms that the signal states a purpose preference, not an access block.
Keep most pages out, but open a public folder	User-Agent: __usage_ai-training; Disallow: /; Allow: /archive/	Check that only intended paths are opened and that the folder contains no confidential material.
Exclude one sensitive folder	User-Agent: __usage_ai-training; Disallow: /archive/	Use this only if the rest of the site should remain allowed under the policy.

Provider timeline: what to do now

Adoption is staggered, so the safest rollout is staged. A site can publish the preference early, then validate how each provider reads it once their support window opens.

Company	Stated support window	Practical implication
Microsoft	Starting February 1, 2027	Sites that want to opt out of training use can prepare the declaration now, before the date arrives.
Google	Early 2027, subject to readiness	Review Google-Extended settings now; if the new preference is used without an explicit Google-Extended rule, Google plans to treat the site as rejecting Google-Extended.
OpenAI	Within the coming months	Run the rule through a robots.txt parser and document the change before the implementation window closes.
Anthropic	Appears to have begun supporting the draft	Check crawl and AI usage logs where available, because early support may precede full ecosystem convergence.

Step-by-step rollout checklist

1. Map every public path that might be used for training: product docs, blog posts, pricing pages, newsrooms, PDFs, API references and downloadable archives.
2. Choose one default position: exclude the site by default and open only low-risk folders, or allow broad training and deny sensitive folders.
3. Add the training preference as a set of rules, then use a robots.txt validator to check parser behavior after each edit.
4. Keep vendor crawler rules separate: if the goal is to stop fetches, continue using existing User-Agent rules for specific crawlers; do not assume the new signal is an access firewall.
5. For Microsoft services, evaluate AI Grounding as a distinct decision; Microsoft's Bing Webmaster Tools guidance treats grounding separately and says it cannot be set in robots.txt.
6. For Google services, inspect Google-Extended separately because it is linked to Gemini model training and some Gemini answer web references.
7. Record the governance rationale: which folders are open, why, who approved them, and when to recheck them after site redesigns, CMS migrations or new content hubs.

Common traps in plain language

• Treating the new token as a universal opt-out. The master draft supports a set of rules for training preference; traditional robots.txt controls may still be needed for crawl blocking.
• Confusing training with retrieval. A model learning from a page is different from a chat system citing a page. Providers use separate controls for those outcomes.
• Overallowing press material. An Allow: /news-kit/ exception can look harmless, but if that directory contains future product roadmaps, case studies or customer examples, it may expose more than intended.
• Ignoring partial adoption. Even after one vendor honors the signal, non-participating systems or already collected copies may not be removed.

FAQ: quick answers for site owners

Does the training declaration prevent AI crawlers from visiting pages?

No. The statement expresses a usage preference. If a site wants to deny access to a particular bot, it must still configure conventional robots.txt rules for that bot.

Is rejecting training the same as rejecting citations?

No. Training and AI answer retrieval are separate use cases. Microsoft provides AI Grounding controls for specific pages within Microsoft services, while Google's behavior around Google-Extended can affect Gemini-related training and reference paths.

What is the lowest-risk move for a new site?

Publish a conservative default, such as excluding the whole site from training while allowing only clearly public folders. Then validate exceptions, review logs after each vendor support date, and treat any change as a content-governance decision, not a one-line configuration trick.

Takeaway from AIdataNewsGlobal

For the General desk, the story is simple: robots.txt is becoming a policy panel for AI use, not only a crawler switch. The immediate action for web teams is to choose a training posture, document the rules, review Google-Extended and Microsoft AI Grounding as separate decisions, and monitor adoption as Google, Microsoft, OpenAI and Anthropic support windows mature.