In May, bots accounted for almost 60 percent of all requests hitting the websites Cloudflare tracks, a threshold chief executive Matthew Prince says he had not expected to reach until the end of the following year. The milestone captures how quickly automated traffic, much of it AI crawlers scraping the web to train models, has overtaken the humans the internet was originally built to serve.
The fuel with no fuel bill
Prince has been blunt about what he thinks needs to change. Companies need to get paid by the AI companies for what is the fuel that runs these AI systems, he has said, arguing that many AI crawlers ingest a site's content to train models that may never send a single reader, or a single cent, back to where the material came from.
A company with a unique vantage point
Cloudflare is positioned to make that case with more authority than most. Its network handles traffic for roughly 20 percent of the entire internet, its customer roster includes Apple, Google and Microsoft, and by the company's own account, around 80 percent of the leading AI companies are Cloudflare customers as well, giving Prince an unusually complete view of just how much of the modern web's traffic is no longer human at all.
Letting sites opt out of training without disappearing from search
Cloudflare has introduced a setting called Disallow AI Training that lets a website stay indexed and discoverable through search while specifically blocking crawlers from using its content to train AI models, separating the question of being found from the question of being harvested. The company has also begun blocking AI training crawlers by default, shifting the burden from publishers having to opt out crawler by crawler to AI companies having to ask for access instead.
Building a marketplace instead of a courtroom
Cloudflare is working to build a marketplace that would let publishers distribute their content directly to AI companies and get compensated for it, effectively formalizing a pay per crawl system rather than leaving each publisher to negotiate its own deal or take a claim to court. Apple, Google and Microsoft have signaled support for the approach, notable given that two of them are themselves building the AI models the policy is meant to constrain. Publishers without that kind of leverage have increasingly turned to litigation instead, with Sony, Warner Music, news organizations and dictionary publishers among those pursuing copyright suits over how their material has been used to train AI systems.
From popularity to credibility
Prince has framed the stakes in broader terms too, arguing that AI is pushing the internet to shift its measure of value away from popularity metrics like pageviews and clicks toward something closer to credibility, and he has floated the idea of an Academy Awards style recognition system for crediting where AI generated answers actually originated. He has projected automated traffic could eventually outstrip human traffic by as much as 1,000 times within five years, a scale he argues makes the question of who pays for all of it impossible to keep postponing.
A web where the readers are mostly machines still needs someone paying the bills. Right now that someone is still the publisher.
Whether Cloudflare's marketplace becomes the standard mechanism for that payment, or simply one option among a growing list of licensing deals and lawsuits, the underlying trend Prince is describing is not in dispute. The web he is describing is one where the majority of visitors are no longer people, and the businesses built to serve people are still waiting to see who is going to pay for what remains.
