The era of free content collection from the web is coming to an end. Here's how new platforms are turning AI bots from non-paying visitors into real customers.
For more than two decades, the Web operated on an unwritten bargain: Google and other search engines indexed publishers' content for free, displayed snippets and summaries on their pages, but in return sent massive traffic back to the original sites. Publishers accepted this "use" of their content because they received visibility, readership and, as a result, advertising revenue.
Google News has perfected this model: it aggregates news from thousands of sources, shows headlines and short excerpts, but each click takes the user to the original site. A win-win balance that has sustained the digital news ecosystem.
Now imagine running a news site and discovering that every day thousands of AI bots scan your articles, use the content to feed their templates, and serve complete responses to users, without ever sending traffic back to your site. AI that doesn't generate links, and doesn't generate clicks: it responds directly. Welcome to the reality of 2025.
According to data from Cloudflare, by mid-2025, 80 percent of AI crawling activity will be for model training, while referrals to publishers (especially from Google) are declining dramatically. The model that has worked for two decades is crumbling-Google itself, with its AI Overviews, is sending less and less traffic to the original sites. The crawl-to-refer ratio shows shocking numbers: Anthropic has a ratio of 38,000 crawls per visitor sent to the site, while OpenAI reaches 1,700:1.
But what if bots could become paying customers instead of just free consumers of content? It's time for a new deal: If AI can no longer guarantee return traffic as Google did, it must at least pay for access to content.
July 2025, Cloudflare announced that it had become the first Internet infrastructure company to block AI crawlers by default, simultaneously introducing the "Pay Per Crawl" system. The news? Every new domain that registers with Cloudflare is now explicitly asked if it wants to allow AI crawlers to access content.
But the real game-changer is the marketplace where publishers can charge fees from AI companies every time one of their pages is scanned. No longer just "allow" or "block," but a third option: "charge."
Even before Cloudflare, TollBit had started building this infrastructure. The startup, which recently raised $24 million in Series A, has created a platform where AI bots can pay websites directly to use their content.
TIME and Adweek are among the first clients who are experimenting with this model, finding that AI represents a new category of customers with specific needs.
According to a detailed analysis, TollBit could already generate about $71 million a year for its publishing partners. Here's how:
Using a typical CPM of $15 for content sites, this translates to $195,000 per day in potential revenue for the TollBit ecosystem, or over $71 million annually.
TollBit suggests two levels of pricing:
The difference can be substantial, making this a particularly attractive model for premium and specialized content.
An analysis by TollBit of 160 websites revealed that AI companies scanned an average of 2 million times in Q4 2024. Each page was scanned an average of 7 times.
Toshit Panigrahi, co-founder of TollBit, explains, "The bot traffic generated by these AI platforms is almost equivalent to the bot traffic of search engines that have been around for 20 years, which is incredible."
DataDome has partnered with both TollBit and other platforms to offer an integrated solution: real-time protection from malicious bots and monetization of compliant ones.
Their data show that AI traffic has tripled in just six months, from 2.6 percent to 8.2 percent of all verified bot traffic.
Skyfire represents a different approach, building an entire payment network for autonomous AI agents. With $8.5 million raised and partnerships with companies like DataDome, Skyfire aims to become the standard payment infrastructure for the AI economy.
Before analyzing the numbers, it is important to note that Pay Per Crawl is not cost-effective for everyone. For sites with limited traffic, allowing free access to AI bots can be strategically more beneficial: it increases discoverability of content in AI systems, builds relationships with emerging platforms, and positions the site as "AI-friendly" for future opportunities.
Direct monetization becomes attractive only when the volumes justify the management complexity and implementation costs.
Prashanth Chandrasekar, CEO of Stack Overflow., sums up the situation perfectly: "Community platforms that power LLMs should be compensated for their contributions so they can reinvest in their communities."
With AI-to-AI commerce projected to reach $46 billion in the next three years, publishers who position themselves early in this ecosystem could find themselves with a significant competitive advantage.
"Pay Per Crawl" is no longer science fiction, but an operational reality. For publishers, the question is not whether this model will take hold, but whether and when they can adapt to capture its value.
La verità scomoda: non tutti dovrebbero monetizzare subito. Per siti con traffico limitato (<50K visite/mese), mantenere accesso libero ai bot AI può essere più vantaggioso strategicamente, costruendo visibilità e relazioni per future opportunità. La soglia di convenienza per la monetizzazione diretta si raggiunge generalmente con volumi di traffico significativi.
Projections show attractive potential revenues for medium and large publishers, especially for quality content and substantial traffic. However, success will depend on the ability to:
For publishers with substantial traffic willing to experiment, 2025 may be the year when AI stops being just a cost and finally becomes a business opportunity. For everyone else, it may be the time to build the foundation to capture this opportunity in the future.
Primary sources:
.svg)
.svg)
.svg)
.jpeg)
.jpeg)
.jpeg)