Cloudflare Arms Website Owners in Fight Against AI Crawlers

Connectivity cloud company Cloudflare is introducing new tools and integrations designed to enable site owners to decide whether automate agents and bots can access their content and add it to artificial intelligence systems.

The new offerings are designed to provide this choice to, for example, businesses that are built on advertising or subscriptions and don’t want AI systems training on their content without compensation, the company said in a Wednesday (July 1) press release.

“We believe that if you’re a business that wants your content in AI systems, we should make it as easy and efficient as possible; but if you’re a business where you do not, then you should have the tools to restrict AI’s access,” Cloudflare said in the release. “In response, Cloudflare is testing new default classifications, delivering deeper insights to customers, making AI search faster, and ensuring creators are compensated when their content powers an answer.”

Cloudflare plans to change its defaults on Sept. 15 to allow for search, but block training and agent use for pages with ads. Customers will be able to change their setting at any time in the dashboard, according to the release.

In addition, the company is introducing a new Attribution Business Insights dashboard that lets businesses see how AI bots consume their content and how much traffic each AI company sends back to the site; testing signals that tell AI crawlers whether a webpage has changed in order to reduce wasted crawling; and evolving its Pay Per Crawl into Pay Per Use so that publishers get paid when their content creates value, per the release.

“Now that the majority of traffic on the internet is nonhuman, we must go further and act faster so that a sustainable ecosystem can emerge,” Cloudflare Co-Founder and CEO Matthew Prince said in the release. “Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent.”

The announcement came a year after Cloudflare introduced a tool that lets website owners decide if they want AI crawlers to access their content, determine how AI firms can use the content, and set a price for access via the Pay Per Crawl model.

Prince said in the Wednesday press release that “we are thrilled with the benefits it has had to the ecosystem.”