How we read AI crawler data

 

A client asked me a question recently that I've been asked a lot this year: "How do you actually know which AI systems are reading our site?" Not why it matters, which Mike and I have covered elsewhere, but how. What are you looking at, and why should we trust it?

It's a fair question, because the answer most people get is a score. A tool puts a sample of questions to ChatGPT and the others, notes which organisations get named, and extrapolates. We've made the case against that elsewhere, so this article is about what we do instead: the method behind the monthly reports we produce for our clients and for our own site.

 

Every website runs on a web server, and every time anything asks that server for a page, the server writes one line to a log file. That line records the time, which page was requested, whether the server was able to return it, the internet address (the IP address) the request came from, and a short piece of text called the user-agent, in which whoever made the request says what they are.

A person's browser sends a user-agent that says, "Chrome on a Mac". Google's crawler sends one that says "Googlebot". When ChatGPT fetches a page from your site to help answer someone's question, it sends one that says "ChatGPT-User".

That last part is the important bit; ChatGPT, Perplexity and Claude all fetch pages live when a person asks them something that your content might answer, and they identify themselves when they do. So your log file contains a record of every time an AI platform has reached for the content on your site.

It's important to note that Google Analytics doesn't see any of this bot activity. GA4 only counts a visit when a browser runs its tracking script and, in the UK, when the visitor has agreed to be tracked. A bot doesn't run scripts and doesn't consent, so it's invisible to "traditional" analytics.

What we're left with is two separate data sources that record different things:

• GA4 records the humans who landed on your website and agreed to be counted.
• The log records everything that touched the server: humans and bots, whether they consented or not.

When we say an AI assistant read a page 2,000 times and sent 15 people to the site, those are two separate counts from two separate sources, and it's the gap between the two numbers that's the finding.

Across the websites we monitor, the logs now record more than three million requests a month from automated visitors of one kind or another, and three in every five of those come from AI systems. On several of those sites, AI reading has overtaken human page views.

 

The first thing to know about log files is that they're enormous! A busy professional-services site generates tens of thousands of log records a day. When we start looking at multiple sites, across the course of a full month, the count quickly ramps up to the millions.

Clearly, nobody's reading that by hand and it's well past what a spreadsheet will open. So before we can start counting anything, the logs have to be collected and put somewhere built for handling data sets of that scale.

For us that's BigQuery, Google's data warehouse:

1. Our client sites send their logs to a log-management service in real time (for error monitoring).
2. We built an automation that pulls the previous day's traffic from there into BigQuery.
3. Once the lines are in, a piece of code reads each record and picks out the data we need.

Getting that working reliably took considerably longer than we expected.

The first hurdle is that log-management tools are built for looking up one request when something has gone wrong, not for handing over a whole day's traffic, and they cap what they'll return in one go.

The second hurdle is spotting errors when working with data on this scale. The first iteration of our data pipeline looked like it was working as intended; however, it was quietly returning only about a third of the records and reporting it as the whole. In the end we built dynamic rules to validate the volume of logs we were interested in analysing against the raw counts, to help flag any suspicious variance.

Once we have the data where we need it, the next step is working out what it means.

 

A month's worth of server log data contains hundreds of different user-agents, and the raw list isn't any use on its own.

Every one has to be matched both to the company operating it and to what it's doing on your site. That's less obvious than it sounds, because each operator runs several bots doing different jobs. We currently catalogue around sixty, and new ones appear most months.

What matters for reading the results from an AI citability perspective is the job each bot is doing, so we group them:
 

This is the part we didn't fully appreciate when we started, and it's the part most likely to catch out anyone doing this for the first time.

A user-agent is just text, and anyone building a scraper bot can make it say whatever they like. 'ChatGPT-User' is a particular favourite, because a lot of sites have been told to let it through. So a raw count of AI user-agents is a strong signal, not a verified one, and a report built on the raw count alone is going to include imposters.

The way to check is the IP address, which is much harder to fake because it's where the request actually came from. In principle that's simple. In practice:

• Some operators publish the addresses their crawlers use, and every request can be checked against them.
• Some don't publish anything, which means their traffic can never be fully verified, however genuine it is.
• The published lists change, so a check that was right last month can be wrong this month.

Doing this properly means keeping those lists current and treating each request as verified, plausible or suspect, rather than as simply AI or not. The headline numbers in every report we produce use only the traffic we can verify or reasonably trust; the suspect traffic is shown separately, rather than counted in with the rest.

If a report you're given doesn't mention verification at all, it's worth asking the question, as it could mean the numbers are quite a way out from the reality. 

 

A few rules run through every query we make. They're the difference between measuring how much load the bots put on your server and measuring how much of your content they actually got, and none of them is complicated on its own. All of them are easy to miss.

• Requests aren't reads: A request is any line in the log. A read is a request where the server actually returned the page. A redirect (the server sent the visitor to a different address, usually because it changed) and a not-found are both requests, but neither is a read, because the visitor didn't get the content. Load figures count every request; reading figures count only the pages that were delivered, so a report that doesn't distinguish them is overstating.
• Today doesn't count: The current day is never complete and drags every average down, so we leave it out.
• Spam isn't AI: Anything probing for login pages or configuration files goes into a separate security view, never into the AI counts.

The 'reading figures' are what we mean when we talk about citability, whether you're the kind of source an AI system can and does reach for. (That's defined properly in our recent article Stop calling it "AI visibility".)

Once the counting rules are applied, we have numbers we can trust. The next question is what they can and can't tell you.
 

A log line proves one thing: an AI system fetched the page. It does not prove that the page was cited, quoted, shown to anyone, or made any difference to the answer.

We describe what the logs show as reading, and we don't upgrade a read to a citation, because the logs can't support that and neither can anything else outside the operators' own systems.

Two things pull the live counts in opposite directions:

• Assistants remember: If five hundred people ask the same question in a week, ChatGPT may fetch your page once and reuse what it found for the other 499, so live fetches undercount how often your content is actually used.
• Not every fetch is discovery: If a person pastes a link to your page into ChatGPT and asks for a summary, that shows up in the log exactly the same as ChatGPT finding your page by itself.

We can't see the question or the person; we can only see the bot. So the most we can say concretely is that live fetches track AI reading in the right direction, not one for one.

Google is a separate scenario entirely:

When Google answers with an AI Overview or in AI Mode, it uses the copy of your site it already holds from ordinary search. There's no separate fetch in the logs and nothing to count. The only first-hand number for that is Google's own, in Search Console's generative AI report, which measures how often your pages appeared in Google's AI answers, not whether a bot visited.

It's a different dataset measuring a different thing, which is why we report it separately rather than mixing it into the log figures, and why we ask clients for Search Console access alongside the logs.

Be clear on how your 'Read to Referral' ratio is constructed:

And the figure people quote most, reads to referrals, isn't really a ratio. Across the sites we monitor it runs at roughly 140 reads for every visit an AI assistant sends through. But the reads come from the logs and the referrals come from GA4, which only counts people who consented and often files an AI-referred visit under "direct". They're two different sources measuring two different things. It's a useful signal for one site over time; it isn't a benchmark to compare across sites, and we don't.

We publish all of that in every report, in a section called "What this doesn't capture". It isn't small print; it's what lets the rest of the report go into a board pack without someone adding caveats later.

 

Once the counts are clean, the view we use most is a journey. Each operator's bots move through your content in stages: train, index, read, represent, refer. Four of the five leave a trace. Here's one operator's month on the example site from our sample report:

Everything above can be done by a competent technical team, and I'd rather people looked at their logs than didn't. There are three things worth keeping in mind, as they will trip you up if you go in without expecting them.

The size: 

A month of logs for a professional-services site is millions of lines, and the tools that store them aren't built to hand them over in bulk. You'll need somewhere to put them that can be queried at that scale, and you'll need to check, against the raw counts, that what you've collected covers everything; ours didn't for a while, and it wasn't easy to spot.

The fakes: 

A meaningful share of what identifies itself as AI traffic isn't. Unless you're checking IP addresses against the operators' published lists, keeping those lists current, and being straight about the operators who don't publish one, your AI figures include traffic that probably isn't AI. 

Knowing what you're looking at:

Requests aren't reads, redirects aren't visits, bulk crawlers aren't assistants, and Google's AI answers don't appear in the logs at all. Each of those is a small distinction that changes the number a lot.