Cloudflare report: Googlebot tops AI crawler traffic
Baku, December 16, AZERTAC
Cloudflare published its sixth annual Year in Review, offering a comprehensive looks at Internet traffic, security, and AI crawler activity across 2025, according to SearchEngineJournal.
The report draws on data from Cloudflare’s network, which spans more than 330 cities across 125 countries and handles over 81 million HTTP requests per second on average.
The AI crawler findings stand out. Googlebot crawled far more web pages than any other AI bot, reflecting Google’s dual-purpose approach to crawling for both search indexing and AI training.
Cloudflare analyzed successful requests for HTML content from leading AI crawlers during October and November 2025. The results showed Googlebot reached 11.6% of unique web pages in the sample.
That’s more than 3 times the pages seen by OpenAI’s GPTBot at 3.6%. It’s nearly 200 times more than PerplexityBot, which crawled just 0.06% of pages.
Bingbot came in third at 2.6%, followed by Meta-ExternalAgent and ClaudeBot at 2.4% each.
The report noted that because Googlebot crawls for both search indexing and AI model training, web publishers face a difficult choice. Blocking Googlebot’s AI training means risking search discoverability.
Throughout 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests across Cloudflare’s customer base. The share fluctuated between 2.4% in early April and 6.4% in late June.
Googlebot alone accounted for 4.5% of HTML requests, slightly more than all other AI bots combined.
The share of human-generated HTML traffic started 2025 at seven percentage points below non-AI bot traffic. By September, human traffic began exceeding non-AI bot traffic on some days. As of December 2, humans generated 47% of HTML requests while non-AI bots generated 44%.
Cloudflare tracks how often AI and search platforms send traffic to sites relative to how often they crawl. A high ratio means heavy crawling without sending users back to source sites.
Anthropic had the highest ratios among AI platforms, ranging from approximately 25,000:1 to 100,000:1 during the second half of the year after stabilizing from earlier volatility.
OpenAI’s ratios reached as high as 3,700:1 in March. Perplexity maintained the lowest ratios among leading AI platforms, generally below 400:1 and under 200:1 from September onward.
Other news in this section
