Google and OpenAI plan one robots.txt line for websites to opt out of AI training

Google and OpenAI said on 30 October 2026 that they plan to honour one shared line in a website's robots.txt file that lets the site opt out of AI training. If you run a business website or an online store, this concerns you: today, keeping your pages out of AI training means writing a separate rule for each AI company. This update explains what was announced, when it may start and what to check on your website this week. It is based on Google's Search Central blog, OpenAI's announcement and the draft published at the Internet Engineering Task Force (IETF), read on 4 November 2026.

What happened: one shared AI training signal for websites

• Google's Search Central blog says Google plans "to support a shared AI training control in robots.txt", so websites can state their AI training preference "across multiple AI providers in a single step".
• OpenAI's announcement says it "will honor a common signal" in robots.txt and calls it a voluntary effort.
• The signal is a "policy token" written in the User-Agent line of robots.txt. The IETF draft, published on 30 October 2026, gives this example: a group that starts with "User-Agent: __usage_ai-training" followed by "Disallow: /". The draft lists its author as working at Anthropic, the company behind Claude.
• Google says that when a website uses this token to opt out, and does not name Google-Extended separately, it plans to treat that as if Google-Extended were blocked. Google-Extended is Google's own AI training control, offered since 2023.
• Both companies say their existing controls keep working. OpenAI's crawler page explains that a site can allow OAI-SearchBot to appear in search results while blocking GPTBot to say its content should not be used for training.
• Timing: OpenAI says it aims to implement the signal "in the coming months". Google says it aims to support it "in 2027".

What is not known: the exact start dates, which other AI companies will follow, and whether the draft will change before it becomes a standard.

Why it matters for a small business

Your website holds work you paid for: service pages, photos, blog posts and the product descriptions you wrote. Until now, saying "do not train on this" meant knowing the name of every AI company's crawler and writing a rule for each one.

One shared line makes that simpler, with a trade-off. Google's post says websites that allow Google-Extended "are eligible to appear in responses and receive clicks" through Gemini Apps. So a broad opt-out may also reduce how often your pages are shown as sources in Gemini answers.

For most small businesses, being found by customers who ask AI tools matters more than keeping a service page out of training. If your content is your product, such as course notes, designs or paid guides, the balance may be different. This is our view, not advice from Google or OpenAI.

The line is also a preference, not a lock. The draft says the token "is not a directive for crawling behavior", so on its own it does not stop search engines from visiting your pages. If you are also deciding what customer information goes into AI tools, our guide on customer data privacy and AI tools covers that side.

Who should care

• Owners with a business website, an online store or a blog.
• People who sell content online: courses, guides, templates, photos or designs.
• Web developers and agencies who manage robots.txt files for clients.
• Marketing teams who want their brand to appear in AI answers.

What changes for your website

Nothing changes on your website unless you or your developer edit robots.txt.

How to check your website's AI settings this week

1. Open your website's robots.txt in a browser: type your address followed by /robots.txt. For example, if your website is example.com, open example.com/robots.txt.
2. Look for the names GPTBot, OAI-SearchBot and Google-Extended. If they do not appear, these companies follow only your general rules, the lines under "User-Agent: *", if there are any.
3. Decide what you want: to be found in AI answers, to keep your content out of AI training, or both. With OpenAI, both are possible, because search and training use different crawlers.
4. Let your developer, or your website builder's support team, make any change. One wrong line, such as "Disallow: /" under "User-Agent: *", can stop search engines from crawling your whole website.
5. Do not rely on the new shared line alone yet. Until Google and OpenAI confirm they have switched it on, the company-specific rules are the ones that work.

If someone else manages your website, you can send them this message:

Hello, Please check our website's robots.txt file and tell me:

1. Which AI crawlers it currently allows or blocks, for example GPTBot, OAI-SearchBot and Google-Extended.
2. Whether we can edit this file ourselves.
3. What would change for us if we used the new shared AI training line once Google and OpenAI support it.

Please do not change anything until we agree. Thank you.

What to watch next

• OpenAI's promised follow-up with details of how it will implement the signal.
• Google's progress updates on the support it plans for 2027.
• Whether other AI companies say they will honour the same line, and whether the IETF draft changes.

Not sure where AI can help your business safely? The free AI Business Checkup takes about three minutes.