Block AI Training in robots.txt for Pakistani Websites: GPTBot, Google-Extended, ClaudeBot and the New Shared Rule
Block AI training on your Pakistani website with GPTBot, Google-Extended and ClaudeBot lines, plus the shared rule Google backed on 30 October, and keep search.
To stop AI companies training on your website today, add separate robots.txt groups for GPTBot (OpenAI), Google-Extended (Google) and ClaudeBot (Anthropic), each with "Disallow: /". On 30 October 2026 Google announced support for one shared rule, User-agent: __usage_ai-training, that will cover every participating AI company in a single line, and OpenAI said the same day that it will honour a shared robots.txt signal. Google says it will honour that rule in 2027, so for now you need both.
This matters for Pakistani publishers, bloggers and shop sites that write their own guides and product text. We read Google's announcement and crawler documentation, OpenAI's crawler page, Anthropic's crawler help page and the new Internet draft on 31 October 2026. The rules below come from those pages, with the one thing nobody should do: block the crawlers that send you search traffic.
Three company rules work today, and the shared rule comes in 2027
Today each AI company reads its own token in robots.txt. Disallow GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. The shared token announced on 30 October 2026 will cover all participating companies at once, but Google's page says support starts in early 2027.
| Company | Token for training | What blocking it does | Affects search? |
|---|---|---|---|
| OpenAI | GPTBot | Content should not be used to train its foundation models | No, OAI-SearchBot is separate |
| Google-Extended | No training of Gemini models, no grounding in Gemini Apps | No effect on Google Search, says Google | |
| Anthropic | ClaudeBot | Future material excluded from training data | No, Claude-SearchBot is separate |
| Shared (draft) | __usage_ai-training | Training preference for all participating providers | Not a crawl rule at all |
The robots.txt lines to block AI training
Put these groups in the robots.txt file at the root of your domain, for example example.pk/robots.txt. Anthropic says to repeat it on every subdomain you want covered. Keep your existing Googlebot and Bingbot rules as they are. These groups only add training preferences.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: __usage_ai-training
Disallow: /The last group is the new shared rule. Adding it now costs nothing, and it starts working as each company adopts it. The draft says a User-agent: * group must not be read as matching the shared token, so a general Disallow does not count as a training opt-out. You have to name it.
You can also allow part of a site. The draft's own example disallows everything but allows a press kit folder, which tells AI companies they may train on that folder only.
Do not block the search crawlers by mistake
Blocking training is different from blocking search. OAI-SearchBot puts your site in ChatGPT search, Claude-SearchBot does the same for Claude, and Googlebot runs Google Search. Block those and you lose visitors. Google says Google-Extended has no effect on Search inclusion or ranking.
OpenAI's crawler page says each setting is independent. A site can allow OAI-SearchBot and disallow GPTBot at the same time. It also says ChatGPT-User, which fetches a page when a user asks about it, is not used for automatic crawling, and robots.txt rules may not apply to it.
- Keep allowed for search: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot.
- Block for training: GPTBot, Google-Extended, ClaudeBot, and the shared token.
- Think before blocking: Claude-User. Anthropic says blocking it stops Claude fetching your page when a user asks.
There is one trade-off on Google's side. Google's blog says websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. Block it, and that route closes. If your business depends on being recommended inside Gemini, weigh that first.
How to add the rules and check them
Many small sites run on WordPress or a hosted builder. Both usually let you edit robots.txt, either through an SEO plugin or the hosting file manager. Anthropic also supports a Crawl-delay line if your server struggles, which it calls a non-standard extension.
- Open yoursite.pk/robots.txt in a browser and copy what is there now.
- Add the four training groups above, below your existing rules.
- Do not touch any Googlebot, Bingbot or OAI-SearchBot group.
- Save, then reload robots.txt in the browser to check the new lines show.
- Run Search Console's robots.txt report to confirm Google can still read it.
If your traffic drops after an edit, the usual cause is a stray Disallow under User-agent: *. The checks to see whether robots.txt or noindex is blocking Google will find it. OpenAI says search-related robots.txt changes can take about 24 hours to take effect.
What robots.txt cannot do
A robots.txt file is a request, not a lock. The draft itself says the shared token is a preference and does not control crawling. It also says private agreements between a site and an AI company can override what robots.txt states.
Blocking also works forward, not backward. Anthropic's page says a ClaudeBot block excludes your site's future material from training. It does not promise to remove what was collected before. For pages that must stay private, use a password, not robots.txt.
And none of this touches the separate question of writing with AI yourself. Google's position on that is in its rules on AI-written content for Pakistani websites, and the extra files some agencies sell are covered in what Google says sites do not need for AI search.
Common questions
Will blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says GPTBot is for training and OAI-SearchBot is for ChatGPT search. Each setting is independent, so you can block one and allow the other.
Does blocking Google-Extended hurt my Google ranking?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
When does the shared __usage_ai-training rule start working?
Google says it plans to support it in early 2027. OpenAI announced on 30 October 2026 that it will honour a shared signal, but its crawler page did not list it when we checked.
Does User-agent: * with Disallow block AI training?
Not under the new draft. It says a wildcard group must not be read as matching the shared training token.
Can I let AI train on some pages only?
Yes. Disallow the whole site, then add Allow lines for the folders you are happy to share, as the draft's press kit example shows.
Who wrote the shared rule?
The Internet draft, dated 30 October 2026, lists Kevin Kelley of Anthropic as author. It is an individual draft, not yet an approved IETF standard.
How we verified this
What we checked, where we read it, and what we could not confirm.
Last checked 31 October 2026. We read Google's Search Central post Supporting a shared AI training control for websites (30 October 2026), Google's common crawlers page (updated 30 October 2026), OpenAI's crawler documentation at developers.openai.com, Anthropic's help article on its crawlers, and the IETF draft "Machine-Readable Training Preference Signal" (draft 00). OpenAI's own 30 October post could not be opened from our connection, so we rely on its published summary, which says OpenAI will honour a shared robots.txt signal for participating providers. We did not test whether any crawler obeys these lines.
About the author
Founder and Editor
Umer Kureshi founded Pakistan Era in July 2024 and runs it as administrator, SEO lead and writer, overseeing website operations, search growth, content strategy and coverage of technology and current affairs.
