For about a month the busiest AI crawler visiting this blog was Bytespider, ByteDance’s crawler. It went from 1,480 requests a week in the first report I ran to 9,312 a month later, which is a lot of attention for a blog about Go tooling from a company that has never once asked me about it.

Mostly it wasn’t Bytespider.

The number I was pleased with

The weekly report counts AI crawlers because a growing share of the people who read anything here never click through to it: an assistant fetches the page, answers the question and moves on, and none of that reaches Search Console. So the report asks Cloudflare how many requests arrived carrying the user agent of a known AI crawler, a day at a time because the analytics on this plan only hand out one-day windows, and adds them up.

The first report said 10,321 requests in a week. A month later it said 18,807. Nearly double, and it’s the kind of chart you don’t look at too hard, because I want AI crawlers reading this blog: I’d rather the models knew my tooling exists when somebody asks one for a CLI framework.

Then I asked a better question of the same data.

A user agent is a claim

A user agent is a string the client types. There’s no certificate on it and nothing signs it, and setting it to GPTBot takes as long as setting it to anything else. I knew that, obviously. I just hadn’t applied it to my own report.

Cloudflare publishes a field that does the checking. A request gets a verified-bot category when it came from the address ranges the operator publishes for that crawler, and nothing when it merely says so. The query was already running, so the change was a handful of lines: ask for that dimension as well, and count the two buckets separately.

The first report after the change, covering 9 to 15 September, had this in it:

More than half of everything calling itself an AI crawler wasn’t one. GPTBot had more impostors than genuine visits that week. Not one request claiming to be CCBot, cohere or Mistral verified. (PerplexityBot didn’t verify either, but Cloudflare stopped vouching for Perplexity’s crawler in 2025, so a zero there says less than it looks.) And Bytespider, the crawler that had nearly doubled my chart, was 515 real requests and thirteen and a half thousand fakes.

My favourite is Google-Extended. That one can’t be genuine at all, because it isn’t a crawler: it’s a token you put in robots.txt to tell Google whether it may use your pages for training, and Google does its crawling as Googlebot. A request wearing the name Google-Extended is wearing a name no legitimate crawler uses.

What the costumes were for

The unverified traffic wasn’t shy about its business. It was asking for .env.bak, .ssh/id_rsa and wp-config.old, which is the standard shopping list of anyone hoping a site has left its secrets lying about. Vulnerability scanning, dressed up as the AI crawlers.

Why those names I can only guess. Plenty of sites wave known crawlers through on the strength of the user agent string alone, and a scanner that looks like a crawler in the logs gets filed under “crawler” by whoever reads them… which is to say, by me, for a month. It’s quite a good disguise if you think about it. I’d said AI crawlers were welcome here, and the costume of choice was an AI crawler.

One number that means one thing

The report counts verified requests as AI traffic now, and lists the rest on their own line as scanners, with the paths they were after. Beside both it prints the zone’s own crawler policy, read out of the bot-management configuration rather than out of my memory of what I set, so the numbers sit next to the rule that shapes them.

The headline figure more than halved the week it changed, and that’s the correct figure. The report after said 19,785 verified, with Bytespider’s impostors still turning up at thirteen thousand a week.

The chart I’d liked was two things added together. One was a crawler count, which is what I thought I was looking at. The other was a measure of how useful those crawler names are to people who aren’t crawlers, which is interesting in its own right and had nothing to do with whether anybody’s assistant was reading my posts. The more the second one grew, the better the chart looked and the less it told me.