Google scrapes news sites nineteen times for every reader it sends back
Cloudflare's Matthew Prince says AI platforms have broken the internet's founding bargain. He's trying to fix it before publishers go extinct
Matthew Prince's company protects a fifth of the internet, so when publishers warned him in 2022 that AI would destroy their industry, he thought he'd heard it before. Media companies always claim the next technology will kill them.
He rolled his eyes. Then he looked at the data.
Ten years ago, Google sent publishers one visitor for every two pages scraped. Today: one visitor per 19 pages. ChatGPT scrapes 1,700 times per visitor. Claude scrapes 73,000 times.
The numbers reveal something extraordinary. For thirty years, the internet ran on a simple bargain: publishers created content, search engines sent traffic back. AI platforms have torn up that deal. They take everything, give almost nothing, and Google—which controls 93% of UK search—has made fighting back impossible.
Prince isn't rolling his eyes anymore. He's waging war.
The bargain that built the internet
The deal was elegant in its simplicity. Publishers created content. Search engines crawled it, learning what existed where. In exchange, they sent traffic back. Both sides won.
This wasn't altruism—it was aligned incentives. Google needed comprehensive, up-to-date content to deliver quality search results. Publishers needed visitors to sell advertising and subscriptions. The better your content, the more traffic Google sent. The more traffic Google sent, the better your content could be.
The internet transformed around this bargain. Text became video. Blogs became social media empires. Individual websites became global publishers. The fundamental exchange never changed: create value, get traffic.
What made it work was reciprocity at scale. Google didn't pay publishers directly, but it compensated them in something equally valuable—attention. Millions of potential readers, delivered daily, who might become subscribers or customers or simply spread your work further.
When ChatGPT launched in November 2022, that reciprocity died.
How AI broke the deal
AI answer engines don't need to send traffic because they've already extracted the value. They consume publisher content, synthesise it into answers, and present those answers directly. The user gets satisfied. The AI gets smarter. The publisher gets nothing.
Watch what happens. Someone asks ChatGPT about mortgage rates. The AI scrapes a financial publisher's detailed analysis, pulls out the key figures, reformulates the explanation, and delivers it in seconds. The user closes ChatGPT, satisfied. They never visit the publisher. They may not even know the publisher exists.
This breaks the reciprocity that made the internet work. Publishers still bear all the costs—reporters, editors, fact-checkers, servers—but lose the compensating traffic. AI platforms reap all the benefits whilst contributing nothing to the ecosystem producing the information they require.
The collapse is measurable. Zero-click searches jumped from 56% to 69% in a single year. When Google shows AI Overviews, users are half as likely to click through. They don't visit competitors instead. They simply stop. Information loop closed, value extracted, publisher ignored.
Traffic declines tell the human story. Digital Content Next, representing the New York Times and Condé Nast, reports median Google referrals fell 10% year-over-year. Non-news publishers dropped 14%. DMG Media saw certain searches decline 89%. Chegg lost half its non-subscriber traffic in twelve months. The Planet D, a travel blog, shut down after traffic fell 90%.
The cruelest part? Publishers cannot fight back. Google bundles its search crawler with its AI crawler. Block the AI scraping and you disappear from search entirely, severing your largest traffic source. It's engineered helplessness—surrender or suicide.
"Publishers are in a bind," says Columbia University's Klaudia Jaźwińska. "Opt out of AI Overviews, opt out of Google Search entirely."
Why Google is the real problem
ChatGPT scrapes at worse ratios than Google. But ChatGPT doesn't control 93% of UK search or 95% of US mobile search. That difference transforms Google from annoyance into existential threat.
Consider the mechanics of Google's advantage. The company scrapes publisher content to train its AI. Uses that content to generate summaries placed above traditional search results. Keeps users on Google rather than sending them to publishers. Then prevents publishers from opting out by bundling its search and AI crawlers together.
This isn't competition. It's a protection racket dressed as innovation.
"When it crawls for search, it's also crawling for AI," Prince explains. "You can't block one without blocking the other. That gives them an inherently unfair advantage."
Google denies the accusation. Liz Reid, vice president of search, insists third-party measurements "inaccurately suggest dramatic declines." When users do click from AI summaries, she argues, they're "higher quality" clicks—readers who stay longer.
Publishers recognise the script. Google has deployed identical rhetoric with every major search change—promises of quality over quantity whilst actual volumes collapse. Helen Havlak, publisher of The Verge, cuts through it: "Google traffic has been declining, and a lot of that decline has lined up pretty clearly with the rise of AI Overviews."
The competitive dynamic is stark. Google extracts value at unprecedented ratios whilst monopoly power prevents publishers from refusing. Danielle Coffey of the News/Media Alliance summarises: "Google uses our content without compensation, offers no way to opt out without disappearing from search entirely, then uses that same content to compete with us."
Most revealing is what Google hasn't done. OpenAI struck licensing deals with publishers. So did Anthropic. Google, which prints money from advertising, has signed exactly one major AI content deal—with Reddit, not a traditional publisher. Why pay when monopoly power means you don't have to?
Prince's three-part strategy
Prince understands that technical solutions cannot defeat monopoly power. His strategy reflects that reality.
Part one: visibility. Cloudflare now shows publishers exactly how often they're being scraped and by whom. You cannot negotiate when you cannot measure.
Part two: leverage. Cloudflare blocks AI crawlers by default for new domains, putting roughly a fifth of the internet behind tools that actually work—unlike the easily-ignored robots.txt protocol that AI companies routinely circumvent. Major publishers including TIME, Condé Nast, and The Associated Press have joined. The message: we're not helpless.
Part three: marketplace. Cloudflare's "Pay Per Crawl" experiment lets publishers charge AI platforms for access. Whether AI companies will participate when they currently get everything free remains uncertain. But the infrastructure exists when regulation forces the issue.
Because regulation is where this fight will be won or lost.
Prince travelled to London in October 2025 to meet the UK's Competition and Markets Authority, which had just designated Google with special status for its "substantial and entrenched" market position. Prince is pushing the CMA to force Google to unbundle its search and AI crawlers—let publishers block one without blocking the other.
"I'm encouraged from conversations with them," Prince says, then adds the implicit threat: "But we have many legislators in many jurisdictions lined up to pass legislation if they don't cooperate."
The regulatory pressure is building. The UK's CMA has special powers over Google. The EU faces antitrust complaints from publishers. US Justice Department antitrust remedies include publisher safeguards. Governments are beginning to grasp what Prince saw in the data: this isn't normal competitive disruption. It's monopoly power weaponised against those who cannot fight back.
Prince's vision extends beyond emergency fixes. He imagines content creators compensated directly by AI companies rather than relying on advertising tied to traffic. A new bargain for a new era. But first, he needs to break Google's stranglehold.
What's really at stake
Prince worries about more than publishers. If AI platforms intermediate every information request, economic power concentrates catastrophically. Two or three companies controlling access to human knowledge is a civilisational risk, not a business problem.
Consider what happens as AI agents proliferate. You ask your agent to find the best restaurant in Soho. It scrapes reviews, synthesises opinions, and delivers a recommendation. You never visit the review sites. Never see the alternatives. Never encounter the small, quirky place that doesn't optimise for AI algorithms.
Now extend that logic. AI agents booking hotels. Comparing insurance. Researching medical treatments. Analysing investments. Every interaction mediated by platforms that extract value whilst contributing nothing to the underlying information ecosystem.
The platforms get smarter. The sources get poorer. Quality deteriorates when creators cannot sustain themselves economically. An AI trained on dying publishers will give progressively worse answers, but users won't know—they never saw the source material.
Then there's the attribution collapse. When 69% of searches end without clicks, information divorces from origin. Readers consume answers without knowing where they came from. Journalists investigate corruption, researchers publish findings, experts share insights—and AI platforms harvest it all whilst the creators remain invisible. The connection between creation and recognition severs.
"This is the most interesting question the world will grapple with over the next five years," Prince says. "I feel like Cassandra, yelling ahead of time that change is coming, and people don't quite understand."
His evolution reveals the threat's subtlety. "Even I was myopic at first. I thought, 'This isn't a problem.' Then, 'This is only a media problem.' Now I'm starting to see it's going to change basically everything."
Whether Prince succeeds depends on governments grasping the pace of change faster than publishers disappear. Traffic keeps declining. AI platforms keep scraping. Publishers keep shutting down. Every closure narrows the information ecosystem, reduces the diversity of voices, eliminates sources of original reporting.
The internet's founding bargain is broken. Prince is betting he can rally regulators quickly enough to forge a new one. But time runs short, and the platforms have momentum.
The question isn't whether the internet will change. It's whether anyone will be left to create the information it distributes.
