CompanyOpenAI

OpenAI adopts shared robots.txt to manage AI training

Original title: Making it easier for websites to express AI training preferences

Plain language

What it is: OpenAI announced it will support a single, shared signal in robots.txt files to let website owners manage their AI training preferences. This allows owners to communicate their choices to multiple AI companies at the same time.
How it works: Instead of writing separate rules for every individual AI crawler, website owners can use one common signal in their robots.txt file. OpenAI plans to implement this signal in the coming months.
Where it is available: This applies to website robots.txt files. It will work alongside OpenAI's existing crawler controls, which will still be respected.
Example: Not stated in the source

What it is: OpenAI will honor a common robots.txt signal developed by the IETF AI Preferences working group to let publishers declare training preferences for multiple AI providers.
How it works: The shared signal reduces the need to maintain separate rules for individual crawlers in robots.txt. OpenAI will implement this signal in the coming months while keeping its existing crawler controls active.
Where it is available: This applies to robots.txt files on web servers.
Example: Not stated in the source

The headline and explanation on this page are based on OpenAI's post. The full details are in the original.

Making it easier for websites to express AI training preferences