Web publishers now have a tool to monetize their content for AI training, thanks to a new open licensing standard called Really Simple Licensing (RSL). On Wednesday, major brands including Reddit, Yahoo, Medium, Quora, and People Inc. announced support for the initiative, which aims to give websites leverage to set payment terms for AI companies scraping their content.
The RSL Standard builds upon the familiar robots.txt protocol, traditionally used to tell web crawlers which parts of a site they can access. But RSL goes further: it allows publishers to embed licensing and royalty terms in their robots.txt files or directly in online books, videos, and training datasets. This means AI developers must pay to crawl or reference content if they want to legally use it for training or generating responses.
Who’s Behind RSL?
The system is spearheaded by the RSL Collective, a newly formed rights organization led by Eckart Walther, co-creator of the RSS standard, and Doug Leeds, former CEO of IAC Publishing and Ask.com. According to Walther, the goal is to create a scalable business model for the web, turning the internet into a place where licensing and compensation rights are clearly defined.
The RSL Standard supports a variety of licensing models, including free access, subscription fees, pay-per-crawl, and pay-per-inference, where websites receive payment whenever an AI references their content to generate a response. Crawlers with other purposes, like search engines or archiving tools, can continue as usual.
Solving the Negotiation Problem
Currently, major media companies such as The New York Times, Vox Media, and News Corp have negotiated individual agreements with AI companies like OpenAI and Amazon. RSL simplifies this process, enabling any publisher to automatically receive compensation without lengthy negotiations.
However, the system’s success depends on AI companies participating. Bots have been accused of ignoring robots.txt files, and tracking inference fees requires compliance from developers. By rallying major publishers, the RSL Collective hopes to make adoption more appealing and enforceable, giving AI developers both a legal and practical incentive to pay.
Implementation and Limitations
While RSL sets licensing terms, it cannot block bots on its own. To enforce access, the RSL Collective is collaborating with Fastly, a content delivery network, which acts like a “bouncer,” only letting in AI bots that have agreed to the licensing terms. Publishers not using Fastly can still request licensing, but cannot yet block non-compliant crawlers until more providers implement similar solutions.
Doug Leeds describes the system metaphorically:
“Fastly is the bouncer at the door to the club, and they won’t let people in unless they have the right ID. RSL is issuing the IDs. So we say, ‘Hey, you’ve agreed to license this content,’ and Fastly says, ‘Come on, in your ID checks out.’”
Looking Ahead
The RSL Standard could mark a turning point in the relationship between web publishers and AI companies, creating a more equitable ecosystem where content creators are fairly compensated for their work. As AI continues to rely on vast datasets scraped from the internet, initiatives like RSL may become critical in defining the economics and ethics of AI training.
