OpenAI Now Honors robots.txt AI Opt-Out, But Only for Nice Bots

What the robots.txt AI opt-out actually does

It turns a long-standing, informal convention into a clearer, shared instruction for AI crawlers.

Robots.txt has always been a gentleman's agreement: a plain text file at the root of a site that tells bots which pages they may crawl. Search engines have followed it for decades. AI companies, until now, have been inconsistent about which directives they read and how they interpret them for training versus indexing.

The new signal, backed by OpenAI, Google, Microsoft and Anthropic, gives those companies a common line to look for. If a site owner adds it, the crawlers are supposed to treat the content as off-limits for training. No legal paperwork, no per-vendor forms, no waiting for a support ticket. Just a few characters in a file that already exists.

That simplicity is the point. Most of the web is run by people who do not have a legal team or a custom CMS plugin. A standard they can paste in is the difference between having a policy and having no policy at all.

The catch: it only stops the honest bots

Robots.txt is a request, not a lock. Anyone willing to ignore it can still scrape your site.

This is the same limitation that has always applied to robots.txt. Well-behaved crawlers from major companies will read it. A scraper run from a random server in another country will not. The agreement raises the floor for responsible actors; it does not raise the ceiling for bad ones.

For site owners, that means the new signal belongs in a layered defense, not as the only defense. Rate limiting, bot detection and blocking known bad IP ranges still matter. The difference is that the big AI companies now have a clear, public way to say yes or no, which makes it easier to hold them to their word.

It also gives publishers something they have lacked: a paper trail. If a crawler from a signatory company ignores the signal, that is now a documented violation of a stated policy, not a gray-area judgment call.

Why OpenAI and rivals agreed now

Pressure from publishers, lawsuits and regulators has made 'we respect robots.txt' a claim worth standardizing.

AI training data has become a legal and reputational minefield. News organizations, authors and image agencies have sued over unauthorized scraping. Regulators in the EU and elsewhere are asking harder questions about consent. A shared opt-out signal is a cheap way for AI companies to show good faith without committing to licensing deals for every site on the internet.

It also helps them operationally. One standard is easier to implement and audit than a dozen vendor-specific meta tags. For Google and Microsoft, which already run large crawl infrastructures, adding a training-specific directive is a small engineering change with a large public-relations payoff.

The timing is not accidental. The more site owners who use the signal, the more the remaining unmarked web looks like fair game. That is a comfortable position for AI companies, and a reminder that opting out is now an active choice rather than a default.

What site owners should do this week

Check your robots.txt, decide your training policy, and write it down.

If you run a blog, a store or a publication, open your robots.txt file and see what is already there. Many sites have rules for search engines but nothing specific to AI training. Adding the shared opt-out line is a five-minute job for most setups, and it is reversible if you change your mind.

Decide deliberately. Blocking training does not block search indexing, so you can keep your Google rankings while refusing to feed model training. If you want to be paid for training data, the signal is a starting point for a conversation, not a substitute for a license.

Document the date you made the change. If a crawler ignores it later, you will want the record.

• Open robots.txt at your site root and check for existing AI directives.
• Add the shared opt-out line if you do not want your content used for training.
• Keep search-engine crawl rules separate so rankings are not affected.
• Log the change date and monitor server logs for AI crawler activity.

What it means for you

The signal is available globally and costs nothing to use. Site owners in the US, UK and EU can add it to robots.txt today; there is no regional rollout or paid tier. Whether a given AI company honors it depends on that company's crawler, not on where your server sits.

For readers who run a personal site or a small business page, this is the first standardized, low-effort way to tell AI trainers to back off. For publishers, it is a useful baseline but not a replacement for legal or licensing strategy.

The robots.txt opt-out is not a magic shield, but it is the first common language between site owners and AI trainers. Use it, document it, and keep watching the logs. The web's default is still 'scrape first'; this signal makes 'ask first' a little easier to choose.