AI Crawler Log Analysis 2026: We Checked 181,474 Hits – 93% of the AI Bots Were Fake

AI crawler log analysis looks trivial: one grep for GPTBot, ClaudeBot or PerplexityBot gives you a number in seconds. That is exactly what practically every guide on the subject recommends — divide the hits by your total line count and you have your “AI bot share”. We did it. Then we checked the number.

The result reversed the entire article: of 54,252 hits claiming to be AI or search crawlers, only 3,909 were real. The remaining 50,343 (92.8%) came from servers that belong to none of the operators whose names they carried in the user agent. The single most active “AI crawler” on our site — 36,146 hits, more than any genuine bot — was 100% fake and did nothing but hunt for passwords and credentials.

This is not an overview article about AI crawlers. It is the analysis of 181,474 real requests to getmind.io across 33 days, every one of them checked against the operators’ officially published IP ranges. All numbers come from our own logs; all verification lists come straight from the vendors.

The short version

Method: Caddy access logs in JSON format, complete from the moment logging was enabled on 2026-08-06. Verification by matching the client IP against the operators’ official prefix lists (retrieved 2026-09-07): claude.com/crawling/bots.json, openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, perplexity.ai/perplexitybot.json, perplexity.ai/perplexity-user.json, plus Google’s googlebot.json and special-crawlers.json.

Why we measured this at all

On 6 August 2026, while writing a different article, we noticed that getmind.io had no access log at all. The log directive was missing from the Caddy block. The very article in which we tell readers to search their server logs for AI crawlers was being written on a server that could not do that.

We added logging that day and stated openly in the article that we could present no crawler numbers, because the measurement had only just begun. Any history would have had to be invented.

That was 33 days ago. The measurement exists now. This is the analysis.

The detour is worth mentioning because it demonstrates the underlying problem: a missing measurement never announces itself. There is no error, no empty view, no red status — only a question nobody asks. We could have claimed for a year that we knew who was reading our site.

Step 1: The naive approach — and what it produces

The standard recipe from every guide looks like this:

grep -Eic 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|CCBot|Bytespider|Amazonbot' access.log

On our data this yields 43,951 AI crawler hits against 21,914 classic search and SEO bots. A nice headline: “AI bots read our site twice as often as Google.”

Broken down, the ranking looked like this:

One thing in this table is wrong, and it is the reason for everything that follows: Claude-SearchBot generated more hits in 16 days than every other bot combined in 33. For a site with roughly 110 articles that makes no sense. A crawler indexing content eventually finishes.

Instead of publishing the number, we opened it up.

Step 2: The question that changed everything — what did the bot request?

Status codes from Claude-SearchBot: {'404': 36146}

All 36,146 hits were 404. Not a single successful fetch. The requested paths:

That is not a crawler. It is a scanner systematically hunting for credentials — AWS keys, Git access, deployment secrets, Firebase credentials — wearing the name of a reputable AI vendor, because that name is exempted from rate limiting and bot defence in a great many configurations.

The second clue sat in the user agent itself. Those 36,130 hits arrived with this string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0; +mailto:support@anthropic.com

Alongside them were 68 hits with this one:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +claudebot@anthropic.com)

The difference is a single character: in the first variant the semicolon after Gecko sits outside the closing parenthesis — the bracket is never closed. Add a mailto: address Anthropic does not use in its bots, and it is clear that whoever forged this rebuilt it from memory.

But a malformed user agent is an indication, not proof. An attacker can fix it tomorrow. The only thing that settles the question is origin.

Step 3: The actual test — the official IP lists

All major operators now publish the IP ranges their crawlers run from. This is the only sound evidence, because the user agent is a field the sender fills in freely — a claim, not an ID card.

Finding Anthropic’s list was the first test in honesty. My guessed addresses (anthropic.com/claudebot.json, anthropic.com/claude-searchbot.json) all returned HTTP 404. A 404 on a guessed URL proves nothing at all — it only says the guess was wrong. Only following the link from the official help page led to the real one:

Anthropic’s documentation states explicitly:

“If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”

Every client IP in our logs was checked against these lists. The result:

Verification by prefix matching (Python ipaddress module, IPv4 and IPv6). For Googlebot, googlebot.json and special-crawlers.json were treated together as the valid set.

Three things stand out:

GPTBot did not visit our site once in 33 days. All 273 hits under that name came from foreign servers. The same is true for Google-Extended and Perplexity-User. Anyone using grep alone would have reported activity for all three that never happened.

The traffic is wrong in both directions. Activity is not only invented — real activity drowns in the forgery. Genuine ClaudeBot: 978 hits, hidden inside 36,499 lines carrying its name.

The share is not small. We are a site with 110 articles and no particular fame. A 92.8% forgery rate is not the outlier of a high-profile target; it is the normal condition of any server with a public IP.

Step 4: The finding that surprised us — the recommended check fails

Search for “verify Googlebot” and you will find the same instruction everywhere, and it comes from Google itself:

“Run a reverse DNS lookup on the accessing IP address from your logs, using the host command. Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”

Then a forward lookup to confirm the name resolves back to the same IP. This is the canonical method, it sits in Google’s official documentation, and it is the basis of countless blog posts and nginx snippets.

We ran it as a control against our IP check. On genuine Googlebot IPs the two methods agreed 10 out of 10 (crawl-66-249-65-193.googlebot.com and so on). Then came the counter-check:

104.155.200.212   212.200.155.104.bc.googleusercontent.com   rDNS confirms Google: True
136.109.197.172   172.197.109.136.bc.googleusercontent.com   rDNS confirms Google: True

Those are two of the IPs our prefix check had flagged as fake. The rDNS method blesses them — because googleusercontent.com is on the accepted list, and because anyone can rent a machine in Google Cloud whose reverse DNS ends in exactly that domain.

We counted it in full:

The busiest of these, 34.19.240.51, sent 2,746 requests on its own — and would pass the universally recommended check as genuine Googlebot. All 11,996 fake Googlebot hits were 404; the most-requested paths were /rclone.conf, /sendgrid.env, /dev/.env and /docker/.env.

🔑 The core of the finding: the instruction is not outdated and not carelessly copied — it is in Google’s own docs. It is correct for a different purpose (Google really does operate services under googleusercontent.com, such as user-triggered fetches), but as bot verification it has a hole the size of a public cloud. A check that accepts a rentable domain as proof of identity verifies the sender’s hosting invoice, not their identity.

Anyone using this method should drop googleusercontent.com and accept only googlebot.com and google.com — or better, check against the JSON prefix lists Google offers in the same document as the “automatic solution”.

Step 5: What the genuine AI crawlers actually read

After removing every forgery, 3,909 verified hits remain. Only here does the analysis get interesting — and it looks nothing like the raw numbers.

ClaudeBot: 978 hits, 67% of them not content at all

659 of 978 hits went to robots.txt and the sitemap. ClaudeBot fetches our rules file 334 times in 33 days — roughly ten times a day — while barely reading any articles. This is a crawler that asks permission very diligently and then does very little.

OAI-SearchBot: 45 hits, 45 of them robots.txt

OpenAI’s search index bot visited on 29 separate days and fetched nothing but robots.txt. Not one article. Not the homepage.

That looked suspicious at first — were we checking wrongly, or blocking it? Our robots.txt in full:

User-agent: *
Allow: /

Sitemap: https://getmind.io/sitemap-index.xml

Everything permitted, for everyone. So the bot asks 45 times for permission that is granted 45 times, and leaves. A crawler that only tests the door and never enters still shows up as “active” in every grep statistic.

PerplexityBot: the only one really reading

Of 771 hits, 678 went to real content, spread broadly across the article base. Of all the bots we checked, PerplexityBot behaves most like a classic search engine crawler.

ChatGPT-User: the most valuable traffic of all

392 verified hits, all status 200, not a single 404 — and exclusively on article pages:

This is the category you least want to overlook. ChatGPT-User is not an index crawler but the fetch that happens when a human has just asked a question and ChatGPT pulls our page as a source. Each of these 392 hits represents a real user query in which our text was cited.

Worth noting: the most-fetched article is the one about AI SEO — and it was retrieved in both language versions.

Step 6: The bigger picture — 64% of our traffic is failures

Step back from crawler names and look at total traffic, and the scale becomes clear:

Almost two thirds of all requests to our server target pages that do not exist. And the search patterns are identical across very different disguises:

Counted requests whose path contains one of: .env, .git, credential, .aws, secret, id_rsa, id_ecdsa, .ssh, service-account, config.json, rclone, actuator, .npmrc, token.

So the same attacker simply swaps name tags. Sometimes an AI crawler, sometimes Googlebot, sometimes a Chrome browser, sometimes not bothering at all and announcing itself as curl. This is precisely why defence by user-agent name cannot work: it filters the one field the attacker can change for free.

What we got wrong in our own measurement

Two errors in our own analysis script are worth reporting, because both looked plausible.

First: our initial run reported for ChatGPT-User “404 share 90%, most frequent path /blog/google-ai-seo/”. That made no sense — the page exists. A live request confirmed HTTP 200. The cause was in the script: I computed the most frequent path across all rows for that bot, while the 404 rate referred only to the fake ones. Two numbers from two different populations stood side by side in one line and read like a single statement. A report that places two metrics next to each other silently claims they describe the same set.

Second: the first attempt to fetch Anthropic’s IP list returned HTTP 404 for three guessed URLs. It would have been convenient to conclude that Anthropic publishes no list — the claim would have sounded plausible and been false. Only the detour via the linked help page produced the real address. A 404 on a URL you invented is a statement about your guess, not about the world.

Both errors surfaced only because the results were cross-checked rather than accepted.

What to do about it in practice

1. Don’t grep — verify

The counting command from the guides overstates the figure on our data by a factor of 14. A sound procedure needs three steps: find the name in the user agent, check the IP against the official list, and only then count.

# 1. Fetch the official lists (refresh regularly!)
curl -s https://claude.com/crawling/bots.json         -o anthropic.json
curl -s https://openai.com/gptbot.json                -o gptbot.json
curl -s https://developers.google.com/static/search/apis/ipranges/googlebot.json -o googlebot.json

import ipaddress, json

def load(path):
    d = json.load(open(path))
    v4 = [ipaddress.ip_network(p['ipv4Prefix']) for p in d['prefixes'] if 'ipv4Prefix' in p]
    v6 = [ipaddress.ip_network(p['ipv6Prefix']) for p in d['prefixes'] if 'ipv6Prefix' in p]
    return v4, v6

def verified(ip, nets):
    addr = ipaddress.ip_address(ip)
    pool = nets[1] if addr.version == 6 else nets[0]
    return any(addr in net for net in pool)

Important: these lists change. Anthropic’s file carried the timestamp 2026-08-18T23:56:36Z when we fetched it. A frozen copy will eventually mark genuine bots as fake — a verifier that goes stale turns from a safeguard into a source of errors.

2. Fix the rDNS method if you use it

If you want to stay with reverse DNS: remove googleusercontent.com from the list of accepted domains. On our data that single change moves detection of fake Googlebot traffic from 0.6% to 100%.

3. Make credential paths lead nowhere

The 31,478 requests for .env and friends correctly returned 404 on our site — the normal case for a statically generated site where no such files exist. For a classic application this is not automatic. If you serve PHP, Node or Python, check whether dot files are reachable by the web server at all.

Our Caddy configuration has a matching rule; the nginx equivalent is well known:

location ~ /\.(?!well-known) { deny all; }

4. robots.txt governs visibility, not security

The verified bots obey the file — ClaudeBot requests it ten times a day. The fake ones fetched it not once. That is the clean division of labour: robots.txt steers the polite ones; against the impolite ones only technical defence helps.

The distinction between bot types matters here. Anthropic describes it in its own documentation:

If you want to refuse training but still be cited, block the training bots (GPTBot, ClaudeBot, Google-Extended) and allow the retrieval and search bots (ChatGPT-User, Claude-User, OAI-SearchBot, PerplexityBot). We currently allow everything — with 392 verified ChatGPT-User fetches, a blanket block would be expensive.

5. Log first, claim second

The most banal point, and the one we failed at ourselves: without an access log, every statement about crawlers is a guess. For Caddy one block suffices:

log {
    output file /var/log/caddy/access.log {
        roll_size 50MiB
        roll_keep 5
        roll_keep_for 720h
    }
}

One warning from experience: caddy validate told us “Valid configuration”, yet the reload ran into a timeout — the log file was owned by root while the service runs as caddy. The website served HTTP 200 throughout, because the old configuration kept running. validate checks the form, not the executability.

Frequently asked questions

How do I identify AI crawlers in my log files?

The first step is searching the user agent for known names (GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, OAI-SearchBot, CCBot, Bytespider, Amazonbot). The decisive second step is checking the client IP against the operators’ official prefix lists. In our analysis of 181,474 requests, 92.8% of all step-one matches were fake — so name matching alone produces a massively inflated number.

Are AI crawlers dangerous for my website?

The verified crawlers are harmless: they respect robots.txt, fetch pages at a moderate pace and create no measurable load on our server. The dangerous ones are the forgeries wearing their names. On our server 31,478 requests specifically hunted for .env files, AWS credentials, SSH keys and deployment secrets — some under the name Claude-SearchBot, some as Googlebot, some as an ordinary Chrome browser.

Can you block GPTBot and ClaudeBot via robots.txt?

Yes. Both operators state they respect robots.txt, and our measurement supports this: the verified bots fetch the file regularly — ClaudeBot alone 334 times in 33 days. Anthropic additionally supports the non-standard Crawl-delay directive. The caveat: a block only affects the genuine bots. The fake ones never fetched our robots.txt even once in 33 days.

What is the difference between ClaudeBot, Claude-User and Claude-SearchBot?

ClaudeBot collects content that may contribute to model training. Claude-User fetches a page when a human is asking Claude a question at that moment. Claude-SearchBot indexes content for web search. Blocking each has a different consequence: with ClaudeBot it concerns training data, with the other two it concerns visibility in answers and search results.

Why is the reverse DNS check for Googlebot unreliable?

Google’s official instruction accepts the domains googlebot.com, google.com and googleusercontent.com. The third belongs to Google Cloud and is available to any paying customer. In our data 37 of 53 fake Googlebot IPs would have passed the check — that is 99.4% of the fake Googlebot traffic. Anyone using the method should drop googleusercontent.com or check against the JSON prefix lists directly.

How much traffic do AI crawlers really cause?

Far less than the raw numbers suggest. Of 181,474 requests across 33 days, 3,909 came from verified crawlers of all vendors combined, and only 2,213 of those hit actual content pages. That is roughly 67 page fetches per day from every verified AI and search bot together. The load does not come from the genuine crawlers but from the scanners impersonating them.

Should I block AI crawlers?

That is a judgement call, not a technical question. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) costs little visibility, because they are not responsible for citations. Blocking retrieval and search bots (ChatGPT-User, Claude-User, OAI-SearchBot, PerplexityBot) makes your site invisible in AI answers. On our site 392 verified hits came from ChatGPT-User — each representing a real user question in which our article was used as a source.

Why does OAI-SearchBot only fetch robots.txt?

We measured it, but we cannot explain it. On 29 of 33 days the bot fetched nothing but /robots.txt — 45 hits, not one article. It is not caused by a block: our robots.txt allows every bot everything. It is possible the file is cached centrally and the actual fetching is done by other infrastructure. We cannot prove that — and a plausible hypothesis is not a measurement.

Do the official IP lists change?

Yes, regularly. Anthropic’s file carried the timestamp 2026-08-18T23:56:36Z when fetched on 2026-09-07, and Google’s lists together contain 589 prefixes. A locally stored copy goes stale and then flags genuine crawlers as fake. Fetching should be automated — a verifier that is never updated eventually inverts its own result.

Conclusion

The starting question was simple: which AI crawlers read our articles? The simple answer — 43,951 hits — was too high by a factor of 14.

What remains once you check origin is a sober picture: 3,909 verified hits in 33 days. ClaudeBot mostly asks for permission. OAI-SearchBot only asks for permission. PerplexityBot actually reads. ChatGPT-User fetched articles 392 times because somebody had just asked a question. GPTBot was not there at all.

But the real finding is not about the crawlers — it is about the measuring. The user agent is a field the sender fills in; counting it means adding up claims. And the verification method recommended everywhere against exactly this problem waved through 99.4% of the forgeries in our data, because it accepts a rentable cloud domain as proof of identity.

Neither of these ever announces itself. A miscounted crawler throws no error, and a verification that lets too much through looks in the log exactly like one that works — in fact it reports “passed” more often.

If you want to reproduce this on your own server, start by checking whether you have an access log at all. We went 33 days without one and would not have noticed without an accident.

Further reading: Google AI SEO covers how visibility in AI answers is changing. IT security vulnerabilities puts the things our scanners were hunting for into context. And setting up a Linux server shows the hardening that stops them finding anything.