Google aur OpenAI ka plan: website ko AI training se opt out karne ke liye robots.txt me ek line

Google aur OpenAI ne 30 October 2026 ko kaha ki woh website ki robots.txt file me ek shared line ko maanenge, jisse website AI training se opt out kar sakti hai (opt out of AI training). Agar aapki business website ya online store hai, to ye aapke kaam ki news hai: aaj apne pages ko AI training se bahar rakhne ke liye har AI company ke liye alag rule likhna padta hai. Is update me dekhiye ki kya announce hua, ye kab shuru ho sakta hai, aur is week apni website par kya check karein. Ye update Google ke Search Central blog, OpenAI ke announcement aur Internet Engineering Task Force (IETF) par publish hue draft par based hai, jo 4 November 2026 ko padhe gaye.

Kya hua: websites ke liye ek shared AI training signal

• Google ke Search Central blog ke hisaab se Google ka plan hai "to support a shared AI training control in robots.txt", taaki websites apni AI training preference "across multiple AI providers in a single step" bata sakein.
• OpenAI ke announcement me likha hai ki woh robots.txt me "will honor a common signal", yaani is common signal ko maanega, aur woh ise ek voluntary effort kehta hai.
• Ye signal ek "policy token" hai jo robots.txt ki User-Agent line me likha jaata hai. 30 October 2026 ko publish hue IETF draft me ye example diya gaya hai: ek group jo "User-Agent: __usage_ai-training" se shuru hota hai, aur uske baad "Disallow: /" aata hai. Draft ke author Anthropic me kaam karte hain, wahi company jo Claude banati hai.
• Google kehta hai ki jab koi website is token se opt out kare, aur Google-Extended ka naam alag se na likhe, to Google ise aise maanega jaise Google-Extended block kiya gaya ho. Google-Extended Google ka apna AI training control hai, jo 2023 se mil raha hai.
• Dono companies kehti hain ki unke purane controls chalte rahenge. OpenAI ke crawler page ke hisaab se ek site OAI-SearchBot ko allow karke search results me aa sakti hai, aur GPTBot ko block karke bata sakti hai ki uska content training ke liye use na kiya jaaye.
• Timing: OpenAI kehta hai ki uska target signal ko "in the coming months" lagu karna hai. Google kehta hai ki uska target ise "in 2027" support karna hai.

Kya pata nahi hai: exact start date, kaun si dusri AI companies isse follow karengi, aur standard banne se pehle draft badlega ya nahi.

Small business ke liye ye kyun important hai

Aapki website me woh kaam hai jiske liye aapne paisa aur time lagaya: service pages, photos, blog posts aur aapke likhe hue product descriptions. Ab tak "is par training mat karo" kehne ke liye har AI company ke crawler ka naam jaanna padta tha aur har ek ke liye rule likhna padta tha.

Ek shared line se ye aasaan ho jaata hai, lekin ek trade-off ke saath. Google ke post ke hisaab se jo websites Google-Extended ko allow karti hain, woh Gemini Apps me "are eligible to appear in responses and receive clicks". Matlab broad opt-out karne se Gemini ke answers me aapke pages source ke roop me kam dikh sakte hain.

Zyada tar small businesses ke liye, AI tools se poochne wale customers ko mil jaana, ek service page ko training se bahar rakhne se zyada important hai. Agar aapka content hi aapka product hai, jaise course notes, designs ya paid guides, to faisla alag ho sakta hai. Ye hamari raay hai, Google ya OpenAI ki advice nahi.

Ye line ek preference hai, taala nahi. Draft kehta hai ki ye token "is not a directive for crawling behavior", yaani akele is line se search engines aapke pages par aana band nahi karte. Agar aap ye bhi decide kar rahe hain ki customer ki kaun si information AI tools me jaaye, to hamara guide customer data privacy and AI tools us side ko cover karta hai.

Kisko dhyan dena chahiye

• Jinke paas business website, online store ya blog hai.
• Jo log online content bechte hain: courses, guides, templates, photos ya designs.
• Web developers aur agencies jo clients ki robots.txt files sambhalte hain.
• Marketing teams jo chahti hain ki unka brand AI answers me dikhe.

Aapki website ke liye kya badlega

Jab tak aap ya aapka developer robots.txt edit nahi karte, aapki website par kuch nahi badlega.

Is week apni website ki AI settings kaise check karein

1. Browser me apni website ki robots.txt kholiye: apna address likhiye aur uske baad /robots.txt. Jaise agar aapki website example.com hai, to example.com/robots.txt kholiye.
2. Usme GPTBot, OAI-SearchBot aur Google-Extended ke naam dhoondhiye. Agar ye naam nahi hain, to ye companies sirf aapke general rules maanti hain, yaani "User-Agent: *" ke neeche wali lines, agar koi hain.
3. Decide kariye ki aapko kya chahiye: AI answers me dikhna, apna content AI training se bahar rakhna, ya dono. OpenAI ke saath dono ho sakte hain, kyunki search aur training ke crawler alag hain.
4. Koi bhi badlav apne developer ya website builder ki support team se hi karwaiye. Ek galat line, jaise "User-Agent: *" ke neeche "Disallow: /", search engines ko aapki poori website crawl karne se rok sakti hai.
5. Abhi sirf nayi shared line par bharosa mat kariye. Jab tak Google aur OpenAI confirm na karein ki unhone ise chalu kar diya hai, tab tak company ke apne rules hi kaam karte hain.

Agar aapki website koi aur sambhalta hai, to aap unhe ye message bhej sakte hain:

Hello, Please check our website's robots.txt file and tell me:

1. Which AI crawlers it currently allows or blocks, for example GPTBot, OAI-SearchBot and Google-Extended.
2. Whether we can edit this file ourselves.
3. What would change for us if we used the new shared AI training line once Google and OpenAI support it.

Please do not change anything until we agree. Thank you.

Aage kya dekhna hai

• OpenAI ka promise kiya hua agla update, jisme signal lagu karne ki details hongi.
• Google ke progress updates, 2027 me support ke plan par.
• Kya dusri AI companies bhi yahi line maanne ki baat kahengi, aur kya IETF draft badlega.

Pata nahi AI aapke business me kahan safely madad kar sakta hai? Free AI Business Checkup me lagbhag teen minute lagte hain.