The AI Training Opt-Out: What Changes in robots.txt in 2027

From 2027, one robots.txt rule will let you opt your website out of AI training across several AI companies at once. On 30 October 2026, Google, Microsoft and OpenAI announced they will honour a shared training preference, based on a proposal written by Anthropic. It replaces the current patchwork, where you have to block each company's training bot by name.

The rule only covers training. It doesn't block crawling and doesn't remove you from search or AI answers, but there are two catches that matter for marketers: Google's version also affects Gemini, and Bing is changing how its old NOARCHIVE and NOCACHE tags work. This guide explains what changes, when, and what we recommend for most brands.

Key takeaways

• One line opts you out of AI training: User-Agent: __usage_ai-training followed by Disallow: /. Google, Microsoft and OpenAI have committed to honour it, starting in late 2026 and early 2027.
• It's a usage preference, not a crawl block. Bots still visit your site, and your pages stay eligible for search and AI answers.
• Google applies it to Gemini grounding too. Opting out through the shared rule also stops Gemini apps from using your content to ground answers, though Google Search and AI Overviews are unaffected.
• Bing users of NOARCHIVE or NOCACHE must act before 1 March 2027. After that date, those tags no longer control AI training or grounding.
• Most brands should not opt out. You want AI models to know your products. Opting out makes sense mainly for publishers and creators whose content is the product.

What's changing at a glance

Sources: Google's crawler documentation, Microsoft Bing announcement, OpenAI announcement, IETF draft.

What the new robots.txt rule looks like

The rule uses a normal robots.txt group with a special name, __usage_ai-training, in place of a bot's name. To opt your whole site out of AI training:

1User-Agent: __usage_ai-training2Disallow: /

You can also opt out everywhere except selected sections. This example allows training only on a press kit folder:

1User-Agent: __usage_ai-training2Disallow: /3Allow: /press-kit/

A few rules are worth knowing:

• The catch-all group doesn't count. A User-Agent: * block for all bots isn't read as a training preference. You need the named group.
• Specific rules win. If you also have a rule for a company's own token, such as Google-Extended, that rule takes priority for that company.
• No group means no preference. If you don't add the group, nothing changes.

What the rule does, and what it doesn't

The shared rule tells AI companies how they may use what they collect. It doesn't stop them collecting it.

It's also not a formal standard yet. The proposal is an individual IETF draft, which means it has no formal standing in the IETF process and could still change. A separate IETF working group is developing a broader approach using a Content-Usage line in robots.txt (for example, Content-Usage: train-ai=n). That approach is still a draft and isn't honoured by the major AI companies today.

The Gemini catch: training and grounding are linked at Google

At Google, the shared opt-out maps to Google-Extended. Google-Extended controls two things together: whether Google uses your content to train future Gemini models, and whether Gemini apps and Vertex AI can use it for grounding, which means pulling in your content at the moment they answer a question.

So if you add the shared opt-out and say nothing about Google-Extended, Gemini apps will stop grounding answers in your pages from early 2027. Your pages will still appear in Google Search, AI Overviews and AI Mode, because those run on Googlebot, not Google-Extended.

You can't split the two at Google. Allowing Google-Extended allows both training and grounding. Blocking it blocks both. With the Gemini app now past 1 billion monthly users, that trade-off deserves a deliberate decision.

If you want to opt out of training at OpenAI and Microsoft but stay available to Gemini, name Google-Extended explicitly:

1User-Agent: __usage_ai-training2Disallow: /3 4User-Agent: Google-Extended5Allow: /

The Bing change: NOARCHIVE and NOCACHE stop controlling AI

Many sites use the NOARCHIVE or NOCACHE meta tags to limit how Bing and Copilot use their content. Microsoft is replacing that with two clear settings, and the old tags stop counting for AI from 1 March 2027.

• AI training: set site-wide through the shared robots.txt rule, or page by page with <meta name="Bingbot" content="ai-training:disallow">.
• AI grounding: set page by page only, with three options. Allow is the default. DisallowExtended keeps your content in Microsoft's own AI experiences, such as Copilot, but excludes third-party grounding services. Disallow keeps it out of generated answers entirely.

These settings don't affect normal Bing search results, but Disallow and DisallowExtended will reduce your visibility in Copilot and other AI answers. If you use NOARCHIVE or NOCACHE today, decide what you want before March, or your pages default to Allow.

Should your brand opt out of AI training?

For most brands, no. When people ask ChatGPT, Gemini or Claude about your category, the model's built-in knowledge shapes the answer, especially when it doesn't search. Opting out of training means future models learn less about your products, pricing and positioning, while competitors who stay open keep shaping that knowledge.

Opting out makes more sense when your content is the product: news publishers, research firms, course creators and other businesses that sell access to what they write. For them, keeping content out of training protects its value, and the shared rule now lets them do it without losing search traffic.

The wider web has already made this split. On Cloudflare's network, 17% of sites restrict AI training, but fewer than 1% block search crawlers. Cloudflare now offers a separate "Disallow AI Training" setting that keeps sites in search results, replacing its older all-in-one AI bot block.

What to do now

1. Check your current robots.txt. Use Lantern's free AI Crawler Checker to see which AI bots you allow and the rule behind each one.
2. Decide your training position. Use the table above. Write the decision down so marketing, legal and engineering agree.
3. Keep search bots open. Whatever you decide on training, allow Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. Our AI crawler list has the full set.
4. Decide on Google-Extended explicitly if you add the shared opt-out, so the Gemini effect is a choice, not an accident.
5. Replace NOARCHIVE and NOCACHE before 1 March 2027 if you use them, with the new Bing settings.
6. Check your CDN settings. CDN and firewall rules can block AI bots regardless of robots.txt.

How to measure the impact

Watch two things after any change. First, your server logs: confirm that search and user-triggered bots still reach your pages. Second, your AI visibility: track whether ChatGPT, Gemini, Copilot and Google's AI features still mention and cite you for your key prompts. A drop in citations after a robots.txt change usually means a rule blocked more than you intended.

Lantern tracks brand mentions and citations across every major AI engine, so you can see the effect of a change within days rather than months.

Our recommendation

Leave AI training switched on unless your content is what you sell. For most marketing teams, being known to AI models is an advantage, and the shared opt-out doesn't change that.

The urgent work is elsewhere. Make sure no rule blocks the search bots that decide whether you appear in AI answers. If you use NOARCHIVE or NOCACHE, replace them before 1 March 2027. And if you do opt out of training, decide on Google-Extended on purpose, because it now carries your Gemini visibility with it.

FAQ

What is __usage_ai-training?

It's a special name used in robots.txt to state a preference about AI training. A group starting with User-Agent: __usage_ai-training and Disallow: / tells companies that honour it not to use your content to train generative AI models.

Does opting out of AI training remove me from ChatGPT or Google AI Overviews?

No. The shared rule covers training only. ChatGPT search, Google Search, AI Overviews and AI Mode use separate crawlers and controls. At Google, it does also stop Gemini apps from grounding answers in your content unless you allow Google-Extended.

When does the AI training opt-out take effect?

Microsoft starts on 1 February 2027, Google in early 2027, and OpenAI "in the coming months" from its October 2026 announcement. Until each company starts, its existing bot rules, such as GPTBot and Google-Extended, still apply.

Do I still need to block GPTBot and Google-Extended?

Those rules keep working and take priority for their company. The shared rule saves you from listing every training bot, but company-specific rules remain the most precise control.

Does the opt-out remove my content from existing AI models?

No. It only affects future training by companies that honour it. Content already used to train existing models stays in those models.

Should a B2B or ecommerce brand opt out of AI training?

Usually not. AI models that know your products describe them more accurately and recommend them more often. Opting out mainly suits businesses that sell their content.