Website owners may soon have a cleaner way to tell several major AI companies not to collect their pages for generative-AI training. Google, OpenAI, Anthropic and Microsoft have committed to honor a proposed new robots.txt directive centered on one shared signal: __usage_ai-training.

That may sound like a tiny punctuation-heavy change reserved for server administrators, but the practical difference is meaningful. Instead of maintaining an ever-growing set of crawler-specific instructions for individual companies, publishers would be able to express a broad preference about AI-training collection in one place. Anthropic has already begun supporting the directive. Microsoft says support will start on February 1, 2027; Google says it will arrive in early 2027; and OpenAI says it plans to implement the signal in the coming months.

The arrangement does not erase content companies may have collected in the past, nor does it stop all web crawling. It is a forward-looking rule for crawlers gathering material for training, and it is separate from the question of whether a page can appear in ordinary web search. Still, common support from four large AI players could make a once-fragmented web-publishing task considerably more manageable.

What the proposed directive does

robots.txt is a plain-text file placed at the root of a website. It communicates crawl preferences to automated agents, commonly called bots or crawlers. A site operator can use it to specify which sections of a site a crawler should avoid and, in some cases, which areas remain accessible.

For generative AI, the problem has been specificity. A publisher that wanted to limit AI-training collection had to identify relevant bots one by one, then write and update separate directives for each provider. The proposed common directive is intended to reduce that administrative burden by giving participating companies one recognizable instruction.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

In this example, the User-Agent line identifies the class of crawler to which the rule applies: bots collecting material for AI training. Disallow: / tells those crawlers not to access the site generally. The slash represents the site’s full path structure, so this is the broadest possible block within that rule.

The Allow line creates a deliberate exception. Here, an operator could block AI-training collection from the rest of the site while permitting access to a press-kit directory. That granularity matters because a site owner may want to make selected official materials available without opening every article, archive page or resource to training crawlers.

It is not a search opt-out

The most important limitation is also the easiest one to miss: this directive is aimed at generative-AI training collection, not crawling in general. It does not inherently prevent a website from being indexed for search, and it does not take a page out of Google results.

Google says it will treat the new signal as a restriction for Google-Extended, the user agent it uses to gather material for Gemini model training. In other words, the instruction concerns that AI-training activity specifically, rather than serving as a universal instruction for every Google crawler or service.

This distinction gives publishers more control than an all-or-nothing switch. A site can potentially preserve search visibility while telling participating companies that its content should not be newly collected for model training. That separation is especially important for organizations that depend on discoverability but still want to set boundaries around AI use.

It also means administrators should read the directive for what it is. Adding it is not a general privacy control, a security control or a guarantee that no automated traffic will ever touch a website. It is an instruction for a particular type of automated collection by companies that recognize and honor the common signal.

Why one rule is a bigger deal than it looks

On a technical level, a shared label is simpler to deploy. On an operational level, it reduces the chance that a publisher’s policy is incomplete simply because a newly relevant crawler was not added to a long list of bot-specific rules. The proposal, written by Anthropic technical lead for standards Kevin Kelley, is under consideration at the Internet Engineering Task Force, an open internet-standards organization.

Standardization is useful here because the preference itself is not complicated: a publisher wants to say yes, no, or perhaps yes only to certain sections. The complexity came from needing to translate that policy separately for each AI company’s crawler. A common directive moves the work from repeated identification of providers toward a single understandable site-level statement.

That does not mean site owners can set it and forget it. The practical value depends on widespread support and on the stated implementation schedules being met. Anthropic’s support is already in place, while the other three companies have given different timelines. A publisher that needs its policy to cover current crawler behavior may still need to understand the provider-specific rules in effect until broad implementation arrives.

The change should also be viewed as a voluntary crawling-control mechanism, rather than a magical shield around online material. The commitment described here is that the participating companies will honor the directive. The rule communicates a preference to compliant crawlers; it is not described as a tool that removes material from the public web or retroactively extracts it from training datasets.

No retroactive reset for previously collected pages

The directive only applies to future collection. If a company already crawled a page before the instruction was in place and before that company supported it, the company already has the page’s content. Adding the new rule cannot undo prior scraping.

That boundary changes how publishers should interpret the news. This is a control over what happens next, not a historical audit, dataset deletion mechanism or rewind button. Its effectiveness grows from the moment it is implemented and respected by the relevant crawler.

For sites publishing frequently, that prospective control can still be significant. A publisher can use it to establish a standing policy for new work without having to repeat a crawler-by-crawler decision every time it posts an article or updates a section. But the timeline matters: each company’s support date defines when its own compatible crawler behavior can begin.

What publishers should check before using it

The basic syntax is short, but the policy decision is not necessarily simple. A site owner should first decide whether the intended rule is a whole-site restriction or a selective permission structure. A blanket block resembles the example’s Disallow: /. A selective approach requires care in naming the directories that should remain available.

  • Decide the scope: determine whether AI-training crawlers should be blocked across the entire site or only from certain sections.
  • Identify intentional exceptions: if a press kit or another area is meant to remain available, state that exception clearly with an Allow rule.
  • Keep the search distinction in mind: this proposed signal is for AI-training collection and does not itself remove pages from Google Search.
  • Account for timing: Anthropic already supports it, while Microsoft, Google and OpenAI have announced later or less specific implementation windows.
  • Do not treat it as retroactive: it cannot recover content collected before the directive applied.

There is also a broader infrastructure lesson: technical controls work best when they map cleanly to a real-world policy. “Do not train on our articles, but you may access our press materials” is a policy a publisher can explain internally. The new directive provides a standardized technical way to express that policy to participating AI-training crawlers.

That kind of clarity is increasingly useful as automated systems multiply. The wider conversation around AI can sometimes drift toward dramatic hypotheticals, while basic cyber hygiene remains essential too; unpatched devices remain a central practical security weakness. The robots.txt proposal addresses a narrower issue: publisher preferences about AI-training scraping.

A modest standard with concrete consequences

The agreement is not a complete answer to every dispute over web data and generative AI. It does not govern past collection, it does not block search indexing, and it does not transform robots.txt into an enforcement system against every bot on the internet. Those limits are crucial.

Yet the proposal is still a substantial usability improvement if the promised support becomes routine. Publishers would no longer need to build their AI-training opt-out around a patchwork of brand-specific crawler names. One clear directive can instead communicate one clear preference to Google, OpenAI, Anthropic and Microsoft.

For website operators, that is the central takeaway: the new signal is targeted, prospective and dependent on compliant implementation, but it offers a more coherent way to manage a question that has become increasingly common. It separates the choice to remain searchable from the choice to permit future generative-AI training collection—and gives publishers a shared language for making that choice.