Governments have largely been ineffective in regulating the unfettered scraping of the web by AI companies. Now, Cloudflare is taking a different approach, tackling the problem from a commercial angle with a beta program that charges AI bots each time they scrape a website. Cloudflare’s CEO, Matthew Prince told Ars Technica:

Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.

Under the program, websites set what can be scraped and what scraping costs. In addition, for new customers, Cloudflare is now blocking AI bots from scraping sites by default, a change from its previous opt-in blocking system.

There are a lot of questions surrounding the viability of Cloudflare’s pay-to-scrape beta, and many details still need to be worked out, not the least of which includes convincing AI companies to cooperate. However, I’m glad to see Cloudflare taking the lead on an approach that attempts to compensate publishers for the value of what AI companies are scraping and put agency back in the hands of creators.