Technology
Logfile analysis is an SEO and GEO goldmine that is too often left untouched - Pieter Serraris
In this episode, Pieter Serraris emphasizes the untapped potential of log file analysis for SEO and generative engine optimization. He discusses how understanding bot behavior can enhance content stra...
Logfile analysis is an SEO and GEO goldmine that is too often left untouched - Pieter Serraris
Technology •
0:00 / 0:00
Interactive Transcript
spk_0
If you're not yet using log files, if you're not yet analyzing them, make sure you start doing it.
spk_0
Just by starting, it's not that complicated. It looks like a lot. It looks like something only technical
spk_0
SEOs can do, but it's really not. Maybe you need some help setting it up at the first time, but there's
spk_0
a lot of cool tools out there that will easily help you. There are a lot of tutorials also everywhere to be found.
spk_0
As soon as it's set up, you will be able to start digging and it's kind of addictive, actually, to start to try to get to know
spk_0
what's happening on your website and start learning from it, start playing with it, start testing.
spk_0
So that's really my key takeaway. Sometimes it will do nothing, whatever you test, but sometimes you really get results out of it.
spk_0
That's what SEO is about, and I don't think you can do SEO without seeing data.
spk_0
And this is really a way of gathering data that's untouched so far for a lot of websites, I think.
spk_0
Hi, I'm Peter Sardaris. This is SEO in 2025, additional insights.
spk_0
Peter, what's your additional insight for SEO in 2025?
spk_0
Well, for me, I think this is going much deeper into log file analysis.
spk_0
I know for quite a lot of SEOs as a log file analysis is already a thing they're doing, but I don't think they're doing much more than scratching the surface, especially if I look around at agencies in my area in the Europe.
spk_0
I think log file analysis is somewhat of a hidden gem for a lot, a gold mine that's not untouched enough.
spk_0
So I know what's what most SEOs link to log file analysis and that's very technical, very technical tool, which of course it is.
spk_0
It is invaluable, especially for bigger websites like e-commerce, like big international multinational websites.
spk_0
It's invaluable to see what's happening to your crawl budget if bots are hitting walls somewhere, if there are a lot of 404s that you cannot find in the traditional SEO tools.
spk_0
It's invaluable. You must do that, especially for bigger websites, but I think there's much more that can be found in these log files.
spk_0
It's really just dig a bit deeper and learn the names of the bots, the user agents that they are using and you can really find out a lot of information, both for SEO purposes, but now also for the generative engine optimization for finding out more what's happening in chat, GPT, in perplexity, etc.
spk_0
Also, these bots will give you your log files, will give you a lot more information.
spk_0
So you'll be able to see if chat GPT visits your websites and by seeing which of chat GPT bots because they actually have at least three that we know of.
spk_0
You will see what the purpose of its visit was. You have the GPT bots, which actually is just the learning bots.
spk_0
Yeah, gathering information to add to the large language model to make it smarter, but there's also, for instance, chat GPT user and whenever you see a hit from that bot, you know actually that your site was being used as a source because it's really real time visits.
spk_0
It's whenever a prompt is given and for which your website is used as a source and chat GPT doesn't have enough information in its LLM in its large language model to answer the prompt.
spk_0
Then the chat GPT user bot will come visit your websites and actually gather more information, more context in order for it to answer the question.
spk_0
So that's really invaluable information that you cannot gather elsewhere for the moment because chat GPT doesn't have an analytics tool or whatever that you can use as an SEO or other marketing expert.
spk_0
So it's very, very important that you can find a lot of information and seeing what these bots are doing is very interesting also.
spk_0
So again, the technical part, gather information if they have troubles, if they hit 404 pages, if they're hallucinating, if they just make up URLs, it also happens.
spk_0
It's also very interesting if that happens because you can learn from that as well.
spk_0
But also for your content strategy, which URLs from your website are being visited by which bot you can learn so much from it, you can use this in your content strategy, use it in your interlinking strategy, etc.
spk_0
So really, really a lot you can learn from the log files.
spk_0
OK, so log file analysis is hyper relevant in the age of AI.
spk_0
It's been relevant for years to add that additional context in terms of what users do on your site and how you can use that to actually augment your SEO strategy.
spk_0
But I mean, with regards to where you find your log files, how you talk to your IT department, I guess, to make sure that the correct information is being retained.
spk_0
So how would you go about summarizing how you find your log files, how you have those conversations with the IT departments and ensuring that the right information is being retained for the right length of time?
spk_0
Yeah, that's from my experience, always a bit of a struggle, a short struggle, but as soon as it's set up, it's OK.
spk_0
But some IT departments have some difficulties with finding the right log files.
spk_0
First of all, it's important that you have the right log files because you have things like add over log files, etc. on your site.
spk_0
That's not relevant for SEO work.
spk_0
What you need is the access log files and they're on your server, on the server of your website.
spk_0
That's why they're so interesting.
spk_0
They're being gathered on a server level.
spk_0
Who you need to talk to is the person in charge of your server, actually.
spk_0
So most of the time that's just your IT department will help you guide the way if it's a different person than just talk to them.
spk_0
Yeah, it always involves some back and forth mailing because the way you can gather these log files, it depends.
spk_0
It's always different depending on which kind of server you're using.
spk_0
So there's not just one easy mail that I can always send to my client's IT departments, but we always get there.
spk_0
It's also a question of what you want to do with the log files.
spk_0
For me, most relevant, if you're able to set up an API with your log files and a third party tool to gather the log files,
spk_0
that's the easiest way that our agency we use Jet Octopus, which is a very cool tool,
spk_0
which sets up the API for you and then links it to Google search, console data,
spk_0
to Google Analytics data, etc.
spk_0
But still, yeah, that's the ideal situation where you have an API because then whenever there's an issue in your log files appearing,
spk_0
then you can immediately get an alert.
spk_0
So that's cool.
spk_0
What you get also asked is for a one-time export of, let's say, at least three weeks of data.
spk_0
That's a very big file you will get then, but there are other tools that you can analyze.
spk_0
The export, for instance, ScreamingFrog has a, besides the well-known ScreamingFrog,
spk_0
you have the ScreamingFrog log file analyzer also.
spk_0
So you can import log file data in there.
spk_0
So that's also very interesting.
spk_0
And then it's just a one-time export you ask it.
spk_0
It can be a good idea if it's just the first step, first getting to know the log files,
spk_0
convincing your clients on an agency side.
spk_0
It might be a good first step.
spk_0
Depending on the type of website you're on, for instance, if you're on a WordPress site,
spk_0
there's also plugins that will actually just do the work for you.
spk_0
We sometimes set up a Dart visitors, which is a WordPress plugin.
spk_0
It's very easy to use, especially if you're, well, it's not just a WordPress plugin.
spk_0
You can use it in other content management systems as well.
spk_0
But for WordPress, it has just a plugin play setup.
spk_0
Just set it up and you will see the bot visits coming in immediately.
spk_0
You can't learn as much from it as having the entire thing coming into a tool like Jettogtapus.
spk_0
But what we have learned from it is quite a lot also.
spk_0
We have it set up for our own website, for instance.
spk_0
And then we see which AI bots are coming in, what Googlebot is doing,
spk_0
which bots are sort of spamming our websites unnecessarily.
spk_0
It's a very easy tool to get to know log files.
spk_0
The setup really, to summarize this long answer,
spk_0
the setup really depends on what you intend to do with it.
spk_0
The ideal situation for me is that you set up a connection between your log files,
spk_0
where it's stored. It gets very technical.
spk_0
You need some sort of book it. It's called where your third party tool can get or it.
spk_0
That's the ideal situation.
spk_0
But you can also ask for an export or just set up a WordPress plugin, for instance.
spk_0
So you talked about AI bots visiting sites and being able to view that in log files.
spk_0
What specifically are you looking for within your log files to analyze that?
spk_0
And also how can you actually piece that together with how your website is performing on the
spk_0
various AI search engines?
spk_0
Yeah, but what we're mostly looking at first thing is which bots are visiting.
spk_0
As I mentioned, for instance, if we take the use case of chat GPT,
spk_0
it's just very interesting to see first for the GPT bot, the sort of the LLM bot,
spk_0
which pages are they using and which aren't today. This already clarifies quite a lot in which
spk_0
either chat GPT finds interesting information or which information is simply not found.
spk_0
So very interesting use case there is to help this bot find more information.
spk_0
Also, obviously, if you're a publisher, the situation is different. Then you probably will not
spk_0
be so happy that GPT bot is visiting your website because it's just basically stealing,
spk_0
borrowing your information in order to get smarter. Then a good use case would be just to use the
spk_0
robot CXT to block this specific bot. What else you can do is, for instance, what we do is looking at
spk_0
the pages that the chat GPT user bot is visiting because this shows that these pages are actually
spk_0
being mentioned in chat GPT. So we think about it. We see this as relevant content. We think about
spk_0
what would the user do if it visits this page. Why is this page mentioned more than others?
spk_0
Is this, for instance, a long read? Is it a very short FAQ page? Again, learning about which type
spk_0
of content is being shown in chat GPT? It's being used as a source. Get also help you out very
spk_0
much because it just gives you a lot of information for your content strategy as well.
spk_0
And what about tying this together with your actual performance in the AI search engines?
spk_0
Actually, learning about your performance in AI engines, for me, it's like puzzle where you
spk_0
have to gather the pieces yourself because we don't have a lot of information in one tool where
spk_0
like in Google ads, for instance, you see everything. You see your impressions, you see your clicks,
spk_0
and you see what people have done on your website. So you don't have that for, for instance,
spk_0
chat GPT. What we do have is a few pieces of the puzzle. We have the log files which shows
spk_0
like your impressions for a part, not all of them, obviously. Only if the LLMs don't have
spk_0
another information about your website for a specific prompt, you will see that you're being
spk_0
used as a source. Let's one piece of the puzzle. You have brand trackers for AI tools where you can see
spk_0
if you automatically push prompts into chat GPT or etc. You will see if you are mentioned,
spk_0
how you are mentioned and maybe which competitors of yours do better or worse than yourself.
spk_0
So that's sort of another piece of the puzzle. Then you have your Google Analytics data
spk_0
where you actually see the visitors coming in. These are the three pieces of the puzzle that we
spk_0
link to each other. You can do that in a look or studio report where you see these pages get,
spk_0
for instance, good impressions because they are mentioned a bit more because they,
spk_0
there we rank a bit higher than our competitors on average. So they give us more clicks eventually.
spk_0
But for me, it's actually, I think learning more about the impressions is just so valuable because
spk_0
the clicks are decreasing. Less people are going to visit your website. So simply learning about
spk_0
how you're being shown, how your website is being used in a very black box tool like
spk_0
chat GPT or perplexity is already a lot of information which you can really use to help steer your,
spk_0
as I mentioned before, your content strategy. But also help these tools fix hallucinations.
spk_0
We've had clients where these tools use a wrong version of the home page, putting a wrong thing
spk_0
behind the URL. It helped us just adding a redirect from the hallucinated home page to the right
spk_0
home page. This helped getting more visibility and getting more clicks to the right home page. So
spk_0
it's just, as I mentioned, it's a very cool way of learning a lot. But the deeper you go, the more you
spk_0
learn, the more you go into the details, sometimes it's really going into a one-time visit of a bot
spk_0
and following it around to see what it does. It's very interesting and you can really learn a lot.
spk_0
And you can sort of communicate with these bots for the first time with these chat GPT's. Like,
spk_0
for instance, if you see a lot of visits to a page on your website that doesn't exist, we had it also.
spk_0
There was, I think, GPT bots that visited. We have an SEO page on our agency website instead of
spk_0
SEO in the directory. Chat GPT kept visiting a page search engine optimization, which was a 404 page.
spk_0
So it visited that page for over 1000 times in one week. So we put a redirect from that page
spk_0
to the right SEO page, let's say. And yeah, it helped us getting found more, getting more visits
spk_0
by sort of communicating to. But it also helped us understand that the full word that's actually
spk_0
chat GPT was using it, a search engine optimization, that it was maybe a keyword or part of a prompt
spk_0
that we needed to mention more instead of only using SEO as an abbreviation. So yeah, a lot of
spk_0
cool use cases to discover. Peter, what's the key takeaway from the future today?
spk_0
Yeah, key takeaway for me is if you're not yet using log files, if you're not yet analyzing them,
spk_0
make sure you start doing it. Just by starting, it's not that complicated. It looks like a lot,
spk_0
it looks like something only technical SEOs can do, but it's really not. Maybe you need some
spk_0
help setting it up at the first time, but there's a lot of cool tools out there that will easily help you.
spk_0
There are a lot of tutorials also everywhere to be found. And as soon as it's set up, you will be able
spk_0
to start digging and it's kind of addictive actually, to try to get to know what's happening
spk_0
on your website and start learning from it, start playing with it, start testing. So that's really
spk_0
my key takeaway. Sometimes it will do nothing, whatever you test, but sometimes you really get
spk_0
results out of it. So that's what SEO is about. And I don't think you can do SEO without seeing data,
spk_0
and this is really a way of gathering data that's untouched so far for a lot of website editing.
spk_0
Hydrosurraris is the SEO leader at OM Collective and you can find no more word at
spk_0
www.wemcollective.com. Thank you very much for adding your additional insight to SEO in 2025.
spk_0
You're welcome. I've been your host Everbane and you've been listening to SEO in 2025
spk_0
Additional Insights. I'm a Gestik series that complements the original podcast, video series and book.
spk_0
Find out more at SEO in 2025. Y'all come.